Skip to content

Commit 084923f

Browse files
authored
Add CI script to check line ends (#1452)
We add CI scripts to ensure that all text files - use UNIX line ends (LF only), and - have a newline character at the end of the file. This PR is inspired by the scripts introduced in the OpenJDK binding in mmtk/mmtk-openjdk#105, but also offers the ability to automatically fix line ends for the user: ```shell ./.github/scripts/ci-check-lineends.sh -f ``` This PR also fixes existing text files that have wrong line ends.
1 parent b6ee146 commit 084923f

16 files changed

Lines changed: 156 additions & 13 deletions

File tree

.github/protected-workflows.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@ events:
1515
anyEvent:
1616
trustAnyone: false
1717
trustCollaborators: false
18-
trustedUserNames: []
18+
trustedUserNames: []

.github/scripts/check-lineends.py

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#!/usr/bin/env python
2+
3+
import argparse
4+
import re
5+
import sys
6+
7+
ANY_NEWLINE = re.compile(rb'(\r\n|\r|\n)')
8+
9+
parser = argparse.ArgumentParser(
10+
prog='check-lineends.py',
11+
description='Line end checker for the MMTk project',
12+
epilog='''
13+
This script checks if the file FILENAME has proper line ends:
14+
15+
1. it uses UNIX line ends, and
16+
2. it has a newline character at the end of the file
17+
18+
If you add the -f option, it will try to fix the line ends if wrong.
19+
''')
20+
21+
parser.add_argument('-q', '--quiet', action='store_true', help='Quiet mode')
22+
parser.add_argument('-v', '--verbose', action='store_true', help='Verbose mode')
23+
parser.add_argument('-f', '--fix', action='store_true', help='Fix files with wrong line ends')
24+
parser.add_argument('filename', nargs='*', help='File name')
25+
26+
verbosity = 1
27+
28+
def pv(level, *args, **kwargs):
29+
if verbosity >= level:
30+
print(*args, **kwargs)
31+
32+
def process_file(filename, fix):
33+
pv(2, "Processing file:", filename)
34+
with open(filename, 'rb') as f:
35+
content = f.read()
36+
37+
non_unix = b'\r' in content
38+
no_eol = not content.endswith(b'\n')
39+
wrong = non_unix or no_eol
40+
41+
if non_unix:
42+
pv(1, "File contains non-UNIX line ends:", filename)
43+
if no_eol:
44+
pv(1, "File does not end with a newline character:", filename)
45+
46+
if wrong and fix:
47+
pv(1, "Fixing file:", filename)
48+
fixed_content = ANY_NEWLINE.sub(b'\n', content)
49+
if no_eol:
50+
fixed_content += b'\n'
51+
with open(filename, 'wb') as f:
52+
f.write(fixed_content)
53+
54+
return wrong
55+
56+
57+
def main():
58+
args = parser.parse_args()
59+
60+
global verbosity
61+
if args.quiet:
62+
verbosity = 0
63+
if args.verbose:
64+
verbosity = 2
65+
66+
any_wrong = False
67+
68+
for filename in args.filename:
69+
if process_file(filename, args.fix) == True:
70+
any_wrong = True
71+
72+
if any_wrong:
73+
sys.exit(1)
74+
75+
if __name__=='__main__':
76+
main()
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
3+
# This is a driver script for check-lineends.py
4+
# It finds text files in the project tree and checks/fixes its line ends.
5+
#
6+
# The CI runs this script during style checking.
7+
#
8+
# Developers may also run this script directly.
9+
# It forwards all command line options to check-lineends.py.
10+
# This means if you add the '-f' option,
11+
# it will automatically fix the line ends of all files we concern.
12+
#
13+
# ./.github/scripts/ci-check-lineends.sh -f
14+
#
15+
# You can also pass the '-v' option to see which files it is checking.
16+
#
17+
# ./.github/scripts/ci-check-lineends.sh -v
18+
#
19+
# In this project, text files use UNIX line ends, and must have a newline character at the end of the file.
20+
# Note that not having a newline character at the end of a file may have unexpected consequences.
21+
# For example, when concatenating multiple files,
22+
# the last line of a file will be joined with the first line of the next file.
23+
# The same may happen when including files using `#include` or `include!` directives in C or Rust.
24+
25+
BAD_LINE_ENDS=0
26+
27+
# TODO: When we introduce the '.gitattributes' file,
28+
# make sure the patterns here matches the patterns in '.gitattributes'.
29+
# Alternatively, find a way to automatically establish the list of files to check
30+
# from the contents of '.gitattributes'.
31+
FILES=$(find . -name 'target' -prune -o -type f -a '(' \
32+
-name '.gitignore' \
33+
-o -name '*.rs' \
34+
-o -name '*.h' \
35+
-o -name '*.yml' \
36+
-o -name '*.sh' \
37+
-o -name '*.toml' \
38+
-o -name '*.lock' \
39+
-o -name '*.py' \
40+
-o -name '*.bt' \
41+
-o -name '*.bt.fragment' \
42+
-o -name '*.md' \
43+
-o -name '*.html' \
44+
-o -name '*.css' \
45+
-o -name '*.js' \
46+
-o -name 'COPYRIGHT' \
47+
-o -name 'LICENSE-*' \
48+
-o -name 'rust-toolchain' \
49+
-o -name '.gitignore' \
50+
')' -print)
51+
52+
if ! xargs $(dirname $0)/check-lineends.py "$@" <<<$FILES; then
53+
BAD_LINE_ENDS=1
54+
fi
55+
56+
57+
if [[ "$BAD_LINE_ENDS" -ne 0 ]]; then
58+
echo "ERROR: Some text files have non-unix line ends or do not have newline character at the end of file."
59+
exit 1
60+
fi

.github/scripts/ci-style.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,13 @@
22

33
export RUSTFLAGS="-D warnings -A unknown-lints"
44

5+
# --- Check line ends of text files ---
6+
7+
if ! $project_root/.github/scripts/ci-check-lineends.sh; then
8+
echo "ERROR: Line ends check failed."
9+
exit 1
10+
fi
11+
512
# --- Check format ---
613
cargo fmt -- --check
714
cargo fmt --manifest-path=macros/Cargo.toml -- --check

CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@ Generally we expect a pull request to meeting the following requirements before
1515
2. The code is well documented.
1616
3. The PR does not introduce unsafe Rust code unless necessary. Whenever introducing unsafe code, the contributor must elaborate why it is necessary.
1717
4. The PR passes the mmtk-core unit tests and complies with the coding style. We have scripts in `.github/scripts` that are used by our Github action to run those checks for each PR.
18-
5. The PR passes all the binding tests. We run benchmarks with bindings to test mmtk-core. A new pull request should not break bindings, as we ensure that our supported bindings always work with the latest mmtk-core. If a pull request makes changes that require the bindings to be updated correspondingly, you can approach the MMTk team on [our Zulip](https://mmtk.zulipchat.com/) and seek help from them to update the bindings.
18+
5. The PR passes all the binding tests. We run benchmarks with bindings to test mmtk-core. A new pull request should not break bindings, as we ensure that our supported bindings always work with the latest mmtk-core. If a pull request makes changes that require the bindings to be updated correspondingly, you can approach the MMTk team on [our Zulip](https://mmtk.zulipchat.com/) and seek help from them to update the bindings.

LICENSE-MIT

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,4 @@ SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
2020
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
2121
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
2222
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
23-
DEALINGS IN THE SOFTWARE.
23+
DEALINGS IN THE SOFTWARE.

docs/userguide/src/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ MMTk is a memory management toolkit providing language implementers with a power
44
and researchers with a multi-runtime platform for memory management research. It is a complete re-write of the original MMTk,
55
which was written in Java as part of Jikes RVM.
66

7-
<iframe width="800" height="600" src="https://www.youtube.com/embed/0mldpiYW1X4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
7+
<iframe width="800" height="600" src="https://www.youtube.com/embed/0mldpiYW1X4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

docs/userguide/src/portingguide/before_start.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,4 @@ Key questions include:
1212
- Does the runtime support precise stack scanning?
1313
- etc.
1414

15-
Thinking through these questions should give you a sense for how big a task a GC port will be.
15+
Thinking through these questions should give you a sense for how big a task a GC port will be.

docs/userguide/src/portingguide/portability.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ The leftmost box should be entirely free of any MMTk-specific code.
2828

2929
> Note: we do currently maintain a fork of OpenJDK which includes some necessary changes to their code base, but this is not MMTk-specific and ideally this will be upstreamed. Our port to V8 is a cleaner example, where we’ve managed to work closely with the V8 team to upstream all of the refactoring of the V8 code base that was necessary for it to support a third party heap.
3030
31-
We structure the code into three repos. Taking the example of the OpenJDK port, the three repos are: the [MMTk core](https://github.com/mmtk/mmtk-core), the [binding repo](https://github.com/mmtk/mmtk-openjdk) containing both parts of the binding, and the OpenJDK repo, which is currently [a fork](https://github.com/mmtk/openjdk) we maintain.
31+
We structure the code into three repos. Taking the example of the OpenJDK port, the three repos are: the [MMTk core](https://github.com/mmtk/mmtk-core), the [binding repo](https://github.com/mmtk/mmtk-openjdk) containing both parts of the binding, and the OpenJDK repo, which is currently [a fork](https://github.com/mmtk/openjdk) we maintain.

docs/userguide/src/portingguide/prefix.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@ This guide is designed to get you started on porting MMTk to a new runtime.
55
We start with an overview of the MMTk approach to porting and then step through recommended strategies for implementing a port.
66

77
There’s no fixed way to implement a new port.
8-
What we outline here is a distillation of best practices that have emerged from community as it has worked through many ports (each at various levels of maturity).
8+
What we outline here is a distillation of best practices that have emerged from community as it has worked through many ports (each at various levels of maturity).

0 commit comments

Comments
 (0)