[Python-projects] simple way to run pylint on many files

[Gelonida]

Hi,


occasionally we'd like to run pylint on quite many files.

If doing this the naive way
(running a python scipt with os.walk calling then python with pylint)
under windows, then quite some time is spent on starting and stopping
new executables.

Is there a simple way of running pylint on many files without having to
start python for each check.



Additionally I would only be interested in the amount of errors, the
amount of warnings and coding style violations and the overal score.


Thanks a lot for any suggestion

[Sylvain Thénault]

Post by Gelonida
Hi,

Hi,

Post by Gelonida
occasionally we'd like to run pylint on quite many files.
If doing this the naive way
(running a python scipt with os.walk calling then python with pylint)
under windows, then quite some time is spent on starting and stopping
new executables.
Is there a simple way of running pylint on many files without having to
start python for each check.
Additionally I would only be interested in the amount of errors, the
amount of warnings and coding style violations and the overal score.

You can easily run pylint programmatically. See pylint.lint.Run class
(its __init__ method actually). You can then easily give a custom
reporter that only display what you're interested in.

[Alexandre Fayolle]

Post by Sylvain Thénault
You can easily run pylint programmatically. See pylint.lint.Run class
(its __init__ method actually). You can then easily give a custom
reporter that only display what you're interested in.

Mind that Pylint is licensed under the GPL. Doing so means that the code
calling Pylint must be licensed under a compatible license.

[Alexandre Fayolle]

Hi Alexandre,
Thanks a lot for pointing this out. It is true one often forgets which
tool is under which license.
I appreciate this.
In my example this wouldn't be an issue
I would just walk through a list of files and report concisely the
Errors/Warnings/Codingstyle issues and the rating.
Nothing I wouldn't be willing to give away.

Sorry if I sounded harsh, this was not my intention. As long as you do not
distribute the code, it is basically a non issue anyway :-)

[Gelonida N]

On 08/03/2010 09:33 AM, Sylvain Th?nault wrote:

Hi Sylvain,

Post by Sylvain Thénault

Post by Gelonida
occasionally we'd like to run pylint on quite many files.
If doing this the naive way
(running a python scipt with os.walk calling then python with pylint)
under windows, then quite some time is spent on starting and stopping
new executables.
Is there a simple way of running pylint on many files without having to
start python for each check.
Additionally I would only be interested in the amount of errors, the
amount of warnings and coding style violations and the overal score.

You can easily run pylint programmatically. See pylint.lint.Run class
(its __init__ method actually). You can then easily give a custom
reporter that only display what you're interested in.

Finally (almost a year later :-( )
I wanted to try using pylint.lint.Run() from within a script in order to
avoid respawning a new python process for each lint run


my current script looks like:

for filename in filenames:
cmd = [ 'pylint', '-f', "text", pm_file)
linter = Popen(cmd, stdout=PIPE, stderr=PIPE)
output = linter.communicate()
exit_code = linter.returncode
do_something_with(output, exitcode)


Now I try to change it to use pylint.lint.Run()
y first test was just performing multiple runs without even trying to
capture the outpout

from pylint import lint
for filename in filenames:
args = [ '-f', "text", pm_file)
lint.Run(args) # first attempt without capturing output

However it seems, that Run exits immediately after the first linting
process.

I found some documentation on the web
http://www.logilab.org/card/pylint_manual
The footline mentions:
card #5560 - latest update on 2011/08/08, . . .

Post by Sylvain Thénault
It is also possible to call Pylint from an other Python program,
thanks to py_run() function in lint module,
assuming Pylint options are stored in pylint_options string, as
from pylint import lint
lint.py_run( pylint_options)
To silently run Pylint on a module_name.py module,
from pylint import lint
(pylint_stdout, pylint_stderr) = lint.py_run( 'module_name.py', True)

However I can't find py_run in lint

What is the correct way to parse multiple python files from the same
executable.

Version info:
pylint 0.24.0,
astng 0.22.0, common 0.56.1
Python 2.6.5 (r265:79063, A
[GCC 4.4.3]

Thanks in advance for any further info.

Ideally I would get the same output as if I were calling pylint from the
command line, because in this case I would not have to cahnge my custom
function do_something_with(output, exitcode)

[Derek Harland]

Post by Gelonida N
Hi Sylvain,

Post by Sylvain Thénault

Post by Gelonida
occasionally we'd like to run pylint on quite many files.
If doing this the naive way
(running a python scipt with os.walk calling then python with pylint)
under windows, then quite some time is spent on starting and stopping
new executables.
Is there a simple way of running pylint on many files without having to
start python for each check.
Additionally I would only be interested in the amount of errors, the
amount of warnings and coding style violations and the overal score.

You can easily run pylint programmatically. See pylint.lint.Run class
(its __init__ method actually). You can then easily give a custom
reporter that only display what you're interested in.

Finally (almost a year later :-( )
I wanted to try using pylint.lint.Run() from within a script in order to
avoid respawning a new python process for each lint run
cmd = [ 'pylint', '-f', "text", pm_file)
linter = Popen(cmd, stdout=PIPE, stderr=PIPE)
output = linter.communicate()
exit_code = linter.returncode
do_something_with(output, exitcode)
Now I try to change it to use pylint.lint.Run()
y first test was just performing multiple runs without even trying to
capture the outpout
from pylint import lint
args = [ '-f', "text", pm_file)
lint.Run(args) # first attempt without capturing output
However it seems, that Run exits immediately after the first linting
process.

Always a good idea to either check the source when you have it ... or at the very least use pydoc ...

$ pydoc pylint.lint.Run
Help on class Run in pylint.lint:

pylint.lint.Run = class Run
| helper class to use as main for pylint :
|
| run(*sys.argv[1:])
|
| Methods defined here:
|
| __init__(self, args, reporter=None, exit=True)

So lint.Run takes an "exit" keyword. Try setting that to False in your script above

lint.Run(Args, exit=False)

d.

[Gelonida N]

Hi Derek,

Thanks a lot.

Post by Derek Harland

Post by Gelonida N
Now I try to change it to use pylint.lint.Run()
y first test was just performing multiple runs without even trying to
capture the outpout
from pylint import lint
args = [ '-f', "text", pm_file)
lint.Run(args) # first attempt without capturing output
However it seems, that Run exits immediately after the first linting
process.

Always a good idea to either check the source when you have it ... or at the very least use pydoc ...

In fact I used pydoc, but unfortunately I used it in the wrong window
(with my distros version of pylint which is only pylint 0.19.0)

Post by Derek Harland
lass Run
|
| run(*sys.argv[1:])
|
|
| __init__(self, args, reporter=None)

So the argument exit did not exist there. that's why I posted the question.

Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find


Running pydoc in the correct window (virtualenv) gives of course the
information, that I was looking for.

Post by Derek Harland
$ pydoc pylint.lint.Run
pylint.lint.Run = class Run
|
| run(*sys.argv[1:])
|
|
| __init__(self, args, reporter=None, exit=True)

Will now have to check, whether there's an elegant way to redirect the
reports output into a string.

Post by Derek Harland
from pylint import lint
(pylint_stdout, pylint_stderr) = lint.py_run( 'module_name.py', True)

Non elegant work around might be to redirect stdout / stderr before
pylint.lint.Run.__init__() is executed


Thanks again




.

[Gelonida N]

On 08/11/2011 09:49 AM, Gelonida N wrote:

Thanks to Derek's answer pylint.lint.Run() doesn't exit anymore.

Post by Gelonida N
from pylint import lint
args = [ '-f', "text", pm_file)
lint.Run(args, exit=False) \

Now I still struggle capturing stdout / stderr of lint.Run()

My first naive approach was

from pylint import lint
from cStringIO import StringIO
from pylint import lint
for filename in filenames:
args = [ '-f', "text", pm_file)
old_stdout = sys.stdout
old_stderr = sys.stderr
sys.stdout = my_stdout = StringIO()
sys.stderr = my_stderr = StringIO()

lint.Run(args, exit=False)

sys.stdout = old_stdout
sys.stderr = old_stderr

output_str = my_stdout.getvalue() + my_stderr.getvalue()
do_something_with(output_str)

However it fails.

I gather 0 characters in output_str and pylint.lin.Run still reports to
the console.

Have to check further.

[Gelonida N]

On 08/11/2011 09:49 AM, Gelonida N wrote:


My next attempt of redirecting pylint's output is
to create a text reporter with output set to a stream object.

Complete code below.

!/usr/bin/env python
filenames = [ __file__ ]
print "Got %d characters" % len(strval)
from pylint import lint
from pylint.reporters.text import TextReporter
from cStringIO import StringIO
args = [ '-f', "text", filename]
my_output = StringIO()
reporter = TextReporter(output=my_output)
#reporter.set_output(my_output)
lint.Run(args, reporter=reporter, exit=False)
output_str = my_output.getvalue()
do_something_with(output_str)

Output still shows up on the console and output_str has a length of 0

What am I doing wrong this time.

[Gelonida N]

I decided to continue the question in another thread as it is actually
another question and as it might easier for other to search.

The new thread is: "How to redirect output of pylint.lint.Run()"

Post by Gelonida N
My next attempt of redirecting pylint's output is
to create a text reporter with output set to a stream object.
Complete code below.

!/usr/bin/env python
filenames = [ __file__ ]
print "Got %d characters" % len(strval)
from pylint import lint
from pylint.reporters.text import TextReporter
from cStringIO import StringIO
args = [ '-f', "text", filename]
my_output = StringIO()
reporter = TextReporter(output=my_output)
#reporter.set_output(my_output)
lint.Run(args, reporter=reporter, exit=False)
output_str = my_output.getvalue()
do_something_with(output_str)

Output still shows up on the console and output_str has a length of 0
What am I doing wrong this time.

[Derek Harland]

Post by Gelonida N
Hi Derek,
Thanks a lot.
...
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

If you grep the source you'll see that py_run now exists in pylint.epylint.

d.

[Gelonida N]

Post by Derek Harland

Post by Gelonida N
...
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

If you grep the source you'll see that py_run now exists in pylint.epylint.

Derek, Thanks once more.

Will check whether pylint.epylint.run_py() is what I would need for my
script or whether the solution is to find out how to redirect output of
pylint.lint.Run.


It still might be useful to fix the official document, don't you think so?

[Derek Harland]

Post by Gelonida N

Post by Derek Harland

Post by Gelonida N
...
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

If you grep the source you'll see that py_run now exists in pylint.epylint.

Derek, Thanks once more.
Will check whether pylint.epylint.run_py() is what I would need for my
script or whether the solution is to find out how to redirect output of
pylint.lint.Run.

I imagine its what you'll want.

Post by Gelonida N
It still might be useful to fix the official document, don't you think so?

I am not a maintainer on this project. But if I may suggest ... as in all open-source projects authored by people who give their valuable time freely and generally have many issues to look at ... if you have found a problem or there is something you would like fixed then the best thing you could do would be to provide an actual patch that fixes that problem.

And in this case the effort required is very small ... the place to start would be in the doc/ subdirectory of the source.

d.

[Gelonida N]

Post by Derek Harland

Post by Gelonida N

Post by Derek Harland

Post by Gelonida N
...
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

If you grep the source you'll see that py_run now exists in pylint.epylint.

Derek, Thanks once more.
It still might be useful to fix the official document, don't you think so?

I am not a maintainer on this project. But if I may suggest ... as in all open-source projects authored by people who give their valuable time freely and generally have many issues to look at ... if you have found a problem or there is something you would like fixed then the best thing you could do would be to provide an actual patch that fixes that problem.
And in this case the effort required is very small ... the place to start would be in the doc/ subdirectory of the source.

Generally speaking your advice is good for the current case I'm at a
complete loss what you mean.

I mentioned, that I looked at the logilab web site (in particular at
http://www.logilab.org/card/pylint_manual#advanced-usage )
and that I found below text.

Post by Derek Harland
It is also possible to call Pylint from an other Python program,
thanks to py_run() function in lint module,
assuming Pylint options are stored in pylint_options string, as
from pylint import lint
lint.py_run( pylint_options)
To silently run Pylint on a module_name.py module,
from pylint import lint
(pylint_stdout, pylint_stderr) = lint.py_run( 'module_name.py', True)

I failed with above instructions and was at the time still struggling to
find a complete solution (still issues with redirecting the output)

additionally grep -R 'grep -R 'To silently run Pylint' $MY_VIRTUALENVDIR

returns only one match, which is in epylint.py
The documentation is however true for epylint so there is nothing to fix.

Downloading the html source from ligilab's web. and editing it (not even
knowing the correct solution) and sending it back as a patch might be a
very small, but useless effort.


I'm very close to a working version and will (as I usually do) follow up
on my own posts.

[Gelonida N]

Post by Derek Harland

Post by Gelonida N

Post by Derek Harland

Post by Gelonida N
...
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

If you grep the source you'll see that py_run now exists in pylint.epylint.

Derek, Thanks once more.
Will check whether pylint.epylint.run_py() is what I would need for my
script or whether the solution is to find out how to redirect output of
pylint.lint.Run.

I imagine its what you'll want.

Just tried it out.
Unfortunately tt's not what I want as the output format is entirely
different from what I get if I run pylint from the command line.

[Gelonida N]

Post by Gelonida N

Post by Derek Harland
Always a good idea to either check the source when you have it ... or at the very least use pydoc ...

In fact I used pydoc, but unfortunately I used it in the wrong window
(with my distros version of pylint which is only pylint 0.19.0)

Inoticed, that if one is using virtualenv, one has to pay attention when
using pydoc. Even if one runs pydoc in a virtualenv window one will get
the wrong documentation.

Pydoc will produce documentation for the packages of the default Python
installation and not documentation from the virtualenv's Python
installation.

A tiny fix for bash users is:

alias pydoc='python `which pydoc`'

[Nicolas Chauvat]

Hi Gelonida,

Post by Gelonida N
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

http://hg.logilab.org/pylint/file/511d09d223c8/doc is the current head
of the mercurial repository. I copied that documentation to the web
site a couple days ago.

If you see mistakes in that documentation:
- please report them to this list,
- or better yet, attach a patch to your report,
- or even better clone http://hg.logilab.org/patches/pylint/, commit your
patch into that repo, make the repo available online and send us a
pull request with your report.

As most of you know, pylint is developed in the open at
http://www.logilab.org/project/pylint where you'll find tickets, bugs,
etc and the patches being worked on are visible at
http://www.logilab.org/project/pylint?tab=vcreview_patches_tab

We have been trying to make it easier for everyone to contribute to
pylint, even occasionnaly, please let us know if we can further improve
things.

Cheers,

[Gelonida N]

Post by Nicolas Chauvat
Hi Gelonida,

Post by Gelonida N
Additionally it might have helped of course if the official web site
(being marked as updated 8th of August 2011) had not mentioned a command
(pylint.lint.run_py), that I couldn't find

http://hg.logilab.org/pylint/file/511d09d223c8/doc is the current head
of the mercurial repository. I copied that documentation to the web
site a couple days ago.
- please report them to this list,

Will do next time.

In a few simple sentences:
The doc does not mention
pylint.lint.Run() as option to rum pylint from within a script.

On the other hand it mentions pylint.int.run_py, which doesn't exist
anymore.

pylint.epylint.run_py() exists, but has a different output format than
running pylint from the command line.

Post by Nicolas Chauvat
- or better yet, attach a patch to your report,

I'm not exactly sure how to best 'fix' the issue.

Probably bewst would be to mention:
pylint.lint.Run() and pylint.epylint.run_py() and explain the intended
use cases.

Further it might be nice to give an example of how to receive pyilints
output as a string.

I posted what I came up with in another thread.
I'm not sure whether this is really the most elegant way.

Post by Nicolas Chauvat
- or even better clone http://hg.logilab.org/patches/pylint/, commit your
patch into that repo, make the repo available online and send us a
pull request with your report.

Well perhaps it's a nice excuse to make first tests with hg. So far I
used only git.

Is there anything like github for hg?
Oherwise I had difficulties to make the repo available online.

Post by Nicolas Chauvat
As most of you know, pylint is developed in the open at
http://www.logilab.org/project/pylint where you'll find tickets, bugs,
etc and the patches being worked on are visible at
http://www.logilab.org/project/pylint?tab=vcreview_patches_tab
We have been trying to make it easier for everyone to contribute to
pylint, even occasionnaly, please let us know if we can further improve
things.

Just a tiny suggestion:

Of course one can google for it, but perhaps this might reduce the
'barrier' for contributors not knowing hg.
A short l document explaining the minimalist flow of
how to checkout the repository (found it in the section source code of
the pylint web site)
how to fix,
how to commit,
how render public a repo
and perhaps one link to a more complete hg doc.

Post by Nicolas Chauvat
Cheers,

Thanks for your answer

[Nicolas Chauvat]

Hi,

Post by Gelonida N
The doc does not mention
...

You chose the option "report to the list" which is more work on our
side than one of the options based on sending a patch, but thank you
for your input, we'll use it as soon as possible.

Post by Gelonida N

Post by Nicolas Chauvat
- or even better clone http://hg.logilab.org/patches/pylint/, commit your
patch into that repo, make the repo available online and send us a
pull request with your report.

Well perhaps it's a nice excuse to make first tests with hg. So far I
used only git.

http://mercurial.selenic.com/wiki/GitConcepts#Command_equivalence_table
http://hg-git.github.com/

Post by Gelonida N
Is there anything like github for hg?
Oherwise I had difficulties to make the repo available online.

https://bitbucket.org/

Post by Gelonida N
Of course one can google for it, but perhaps this might reduce the
'barrier' for contributors not knowing hg.
A short l document explaining the minimalist flow of
how to checkout the repository (found it in the section source code of
the pylint web site)
how to fix,
how to commit,
how render public a repo
and perhaps one link to a more complete hg doc.

http://mercurial.selenic.com/quickstart/
http://mercurial.selenic.com/wiki/Tutorial
http://hginit.com/