[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Pki-devel] [PATCH] 71-2 Patch for ticket 316- Adding pylint to the build process



PFA the patch  which addressed all comments mentioned below.
Raised a ticket for question number 8.

--Abhishek

On Fri, 2013-07-26 at 16:13 -0400, Abhishek Koneru wrote:
> A few questions/ comments:
> 
> 1. In your build, you create a pylintscan directory within the source
> tree under pki.  I would prefer this to take place in the build area so
> that it does not clutter up your source tree.  The pylintscan is a build
> artifact.
> 
> -- Sure can be moved. I can delete the folder once the scan is done. since the scan actually takes place before
>    the build. (As per answer 2 - can i still keep pylintscan in pki?)
> 
> 2. Do you need to remove the directory prior to each run?
> 
> -- Not required. I will remove it in the script. Will just leave the pylint-report.
>    Is it ok to put the report in the root directory of the code tree - pki?
> 
> 3. What parts of the pylintrc are not default? ie. which parts did you
> change?
> 
> -- The disable-ids property. (Will not disable conventions [C] and refactor[R].)
> -- Max lines - from 80 to 100 
> -- Removed 'string' package from deprecated modules.
> -- Allowing variables to be named 'rv'. (property - good-names)
> -- Set property - include ids = yes.
> 
> 4. I noticed that you added JSONEncoder to the pylintrc as a class for
> which private members should not be checked.  Does that allow you to
> eliminate the comment to # pylint: disable-msg=E0202 in encoder.py?
> 
> -- Forgot to squash a few final changes made to pylintrc. it was actually not included in pylintrc.
> 
> 5. Can you explain the rationale for ignoring  W0511 and W0105?
>  W0105 (pointless-string-statement):
>        String statement has no effect Used when a string is used as a statement (which of course has no effect). This is a particular case of W0104 with its own message so you can easily disable it if you’re   #       using those strings as documentation, instead of comments.
>  W0511 (fixme): Used when a warning note as FIXME or XXX is detected.
> 
> -- Will add the same to the comments for disabled ids.
> 
> 6. I would prefer to not ignore Convention, refactor etc. and to have
> these show up in the pylint reports.  Based on the return code for
> pylint, you can have the build fail only if the bits for errors and
> warnings are set.
> 
> -- Can be done. will make the change.
> 
> 7. The pylint command line seems to have all the python executables
> listed -- is this required?  It means having to update this list every
> time an executable is added, which is a step that can easily be missed.
> Can you just point the scan to the top level directory?  
> 
>  -- Yes. Did not notice that Endi added __init__.py in all directories. Will change that.
> 
> 8.  Similarly to the point above, is there a way to scan the source tree
> to find all the python code?
> 
>  -- Yes it can be done. But since the number of source files is very small, i used this approach of copying
>     the directories directly instead of scanning all the code.
> 
> On Mon, 2013-07-22 at 21:32 -0400, Abhishek Koneru wrote:
> > Please review the patch which adds a script and also the pylint
> > configuration file to the code tree. The script is called in the compose
> > script for core packages before the actual packaging is done. If any
> > errors or warnings are reported by pylint, the build fails.
> > 
> > I did not add pylint as part of build-requires in the spec file for
> > pki-core, but have put a check in the script to bypass trying to scan if
> > pylint is not installed but with a comment stating the same in the log.
> > 
> > --Abhishek
> > _______________________________________________
> > Pki-devel mailing list
> > Pki-devel redhat com
> > https://www.redhat.com/mailman/listinfo/pki-devel
> 
> 
> 
> _______________________________________________
> Pki-devel mailing list
> Pki-devel redhat com
> https://www.redhat.com/mailman/listinfo/pki-devel

>From 13a2a254fcfc9526aff088447ba5a5b7ac531402 Mon Sep 17 00:00:00 2001
From: Abhishek Koneru <akoneru redhat com>
Date: Mon, 22 Jul 2013 21:16:59 -0400
Subject: [PATCH] Adding pylint scan to the build.

Adding the scripts and the pylintrc file to the build process.
The script is called before packaging the classes in the compose
scripts.

Ticket #316
---
 dogtag.pylintrc                   |  262 +++++++++++++++++++++++++++++++++++++
 pylint-build-scan.sh              |   69 ++++++++++
 scripts/compose_pki_core_packages |   16 +++
 3 files changed, 347 insertions(+)
 create mode 100644 dogtag.pylintrc
 create mode 100755 pylint-build-scan.sh

diff --git a/dogtag.pylintrc b/dogtag.pylintrc
new file mode 100644
index 0000000000000000000000000000000000000000..aeb0a2362a4d0741b8824e4cf10b68a22bc3c4bb
--- /dev/null
+++ b/dogtag.pylintrc
@@ -0,0 +1,262 @@
+[MASTER]
+
+# Specify a configuration file.
+#rcfile=
+
+# Python code to execute, usually for sys.path manipulation such as
+# pygtk.require().
+#init-hook=
+
+# Profiled execution.
+profile=no
+
+# Add files or directories to the blacklist. They should be base names, not
+# paths.
+ignore=CVS
+
+# Pickle collected data for later comparisons.
+persistent=yes
+
+# List of plugins (as comma separated values of python modules names) to load,
+# usually to register additional checkers.
+load-plugins=
+
+
+[MESSAGES CONTROL]
+
+# Enable the message, report, category or checker with the given id(s). You can
+# either give multiple identifier separated by comma (,) or put this option
+# multiple time.
+#enable=
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifier separated by comma (,) or put this option
+# multiple time (only on the command line, not in the configuration file where
+# it should appear only once).
+# W0105 (pointless-string-statement):
+#  	String statement has no effect Used when a string is used as a statement (which of course has no effect). This is a particular case of W0104 with its own message so you can easily disable it if you’re   #       using those strings as documentation, instead of comments.
+# W0511 (fixme): Used when a warning note as FIXME or XXX is detected.
+disable=W0511,W0105
+
+
+[REPORTS]
+
+# Set the output format. Available formats are text, parseable, colorized, msvs
+# (visual studio) and html. You can also give a reporter class, eg
+# mypackage.mymodule.MyReporterClass.
+output-format=text
+
+# Include message's id in output
+include-ids=yes
+
+# Include symbolic ids of messages in output
+symbols=no
+
+# Put messages in a separate file for each module / package specified on the
+# command line instead of printing them on stdout. Reports (if any) will be
+# written in a file name "pylint_global.[txt|html]".
+files-output=no
+
+# Tells whether to display a full report or only the messages
+reports=yes
+
+# Python expression which should return a note less than 10 (10 is the highest
+# note). You have access to the variables errors warning, statement which
+# respectively contain the number of errors / warnings messages and the total
+# number of statements analyzed. This is used by the global evaluation report
+# (RP0004).
+evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
+
+# Add a comment according to your evaluation note. This is used by the global
+# evaluation report (RP0004).
+comment=no
+
+
+[FORMAT]
+
+# Maximum number of characters on a single line.
+max-line-length=100
+
+# Maximum number of lines in a module
+max-module-lines=1000
+
+# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
+# tab).
+indent-string='    '
+
+
+[MISCELLANEOUS]
+
+# List of note tags to take in consideration, separated by a comma.
+notes=FIXME,XXX,TODO
+
+
+[VARIABLES]
+
+# Tells whether we should check for unused import in __init__ files.
+init-import=no
+
+# A regular expression matching the beginning of the name of dummy variables
+# (i.e. not used).
+dummy-variables-rgx=_|dummy
+
+# List of additional names supposed to be defined in builtins. Remember that
+# you should avoid to define new builtins when possible.
+additional-builtins=
+
+
+[SIMILARITIES]
+
+# Minimum lines number of a similarity.
+min-similarity-lines=4
+
+# Ignore comments when computing similarities.
+ignore-comments=yes
+
+# Ignore docstrings when computing similarities.
+ignore-docstrings=yes
+
+# Ignore imports when computing similarities.
+ignore-imports=no
+
+
+[TYPECHECK]
+
+# Tells whether missing members accessed in mixin class should be ignored. A
+# mixin class is detected if its name ends with "mixin" (case insensitive).
+ignore-mixin-members=yes
+
+# List of classes names for which member attributes should not be checked
+# (useful for classes with attributes dynamically set).
+ignored-classes=SQLObject
+
+# When zope mode is activated, add a predefined set of Zope acquired attributes
+# to generated-members.
+zope=no
+
+# List of members which are set dynamically and missed by pylint inference
+# system, and so shouldn't trigger E0201 when accessed. Python regular
+# expressions are accepted.
+generated-members=REQUEST,acl_users,aq_parent
+
+
+[BASIC]
+
+# Required attributes for module, separated by a comma
+required-attributes=
+
+# List of builtins function names that should not be used, separated by a comma
+bad-functions=map,filter,apply,input
+
+# Regular expression which should only match correct module names
+module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
+
+# Regular expression which should only match correct module level names
+const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$
+
+# Regular expression which should only match correct class names
+class-rgx=[A-Z_][a-zA-Z0-9]+$
+
+# Regular expression which should only match correct function names
+function-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct method names
+method-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct instance attribute names
+attr-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct argument names
+argument-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct variable names
+variable-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct list comprehension /
+# generator expression variable names
+inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
+
+# Good variable names which should always be accepted, separated by a comma
+good-names=i,j,k,ex,Run,rv,_
+
+# Bad variable names which should always be refused, separated by a comma
+bad-names=foo,bar,baz,toto,tutu,tata
+
+# Regular expression which should only match functions or classes name which do
+# not require a docstring
+no-docstring-rgx=__.*__
+
+
+[DESIGN]
+
+# Maximum number of arguments for function / method
+max-args=5
+
+# Argument names that match this expression will be ignored. Default to name
+# with leading underscore
+ignored-argument-names=_.*
+
+# Maximum number of locals for function / method body
+max-locals=15
+
+# Maximum number of return / yield for function / method body
+max-returns=6
+
+# Maximum number of branch for function / method body
+max-branchs=12
+
+# Maximum number of statements in function / method body
+max-statements=50
+
+# Maximum number of parents for a class (see R0901).
+max-parents=7
+
+# Maximum number of attributes for a class (see R0902).
+max-attributes=7
+
+# Minimum number of public methods for a class (see R0903).
+min-public-methods=2
+
+# Maximum number of public methods for a class (see R0904).
+max-public-methods=20
+
+
+[CLASSES]
+
+# List of interface methods to ignore, separated by a comma. This is used for
+# instance to not check methods defines in Zope's Interface base class.
+ignore-iface-methods=isImplementedBy,deferred,extends,names,namesAndDescriptions,queryDescriptionFor,getBases,getDescriptionFor,getDoc,getName,getTaggedValue,getTaggedValueTags,isEqualOrExtendedBy,setTaggedValue,isImplementedByInstancesOf,adaptWith,is_implemented_by
+
+# List of method names used to declare (i.e. assign) instance attributes.
+defining-attr-methods=__init__,__new__,setUp
+
+# List of valid names for the first argument in a class method.
+valid-classmethod-first-arg=cls
+
+# List of valid names for the first argument in a metaclass class method.
+valid-metaclass-classmethod-first-arg=mcs
+
+
+[IMPORTS]
+
+# Deprecated modules which should not be used, separated by a comma
+deprecated-modules=regsub,TERMIOS,Bastion,rexec
+
+# Create a graph of every (i.e. internal and external) dependencies in the
+# given file (report RP0402 must not be disabled)
+import-graph=
+
+# Create a graph of external dependencies in the given file (report RP0402 must
+# not be disabled)
+ext-import-graph=
+
+# Create a graph of internal dependencies in the given file (report RP0402 must
+# not be disabled)
+int-import-graph=
+
+
+[EXCEPTIONS]
+
+# Exceptions that will emit a warning when being caught. Defaults to
+# "Exception"
+overgeneral-exceptions=Exception
diff --git a/pylint-build-scan.sh b/pylint-build-scan.sh
new file mode 100755
index 0000000000000000000000000000000000000000..f512cd636d7752ff9513fa0e7fda767a0bb5ce98
--- /dev/null
+++ b/pylint-build-scan.sh
@@ -0,0 +1,69 @@
+#!/bin/sh -x
+
+if [ -z "$1" ]
+then
+  echo "PKI codebase home not specified. Could not scan the python scripts. Returning 0 - for SUCCESS"
+  echo 0
+  exit 0
+fi
+
+rv=`rpm -qa pylint`
+
+if [ -z $rv ]
+then
+  echo "Pylint not installed. Could not perforn scan. Returning 0 - for success"
+  echo 0
+  exit 0
+fi
+
+HOME_DIR=$1
+
+PYLINT_TEMP_DIR="$HOME_DIR/../packages/pylintscan/pki"
+
+PYLINT_RC_FILE_PATH="$HOME_DIR/dogtag.pylintrc"
+
+if [ -d "$PYLINT_TEMP_DIR/.." ]
+then
+  rm -rf "$PYLINT_TEMP_DIR/.."
+fi
+
+cd $HOME_DIR
+mkdir -p $PYLINT_TEMP_DIR
+
+# Copy all the python code to their respective directories
+# Same as the deployed structure
+
+cp -r base/server/python/pki/server/ $PYLINT_TEMP_DIR
+
+cp -r base/common/python/pki/*.py $PYLINT_TEMP_DIR
+
+cp -r base/server/sbin $PYLINT_TEMP_DIR
+
+cp -r base/common/sbin/* $PYLINT_TEMP_DIR/sbin
+
+cd $PYLINT_TEMP_DIR/..
+
+rv=`pylint --rcfile=$PYLINT_RC_FILE_PATH pki/ pki/sbin/pkispawn pki/sbin/pkidestroy pki/sbin/pki-upgrade pki/sbin/pki-server-upgrade >> pylint-report`
+
+status=$?
+
+#Excerpt from pylint man page
+#OUTPUT STATUS CODE
+#       Pylint should leave with following status code:
+#           * 0 if everything went fine
+#           * 1 if a fatal message was issued
+#           * 2 if an error message was issued
+#           * 4 if a warning message was issued
+#           * 8 if a refactor message was issued
+#           * 16 if a convention message was issued
+#           * 32 on usage error
+#
+#       status 1 to 16 will be bit-ORed so you can know which different categories has been issued by analysing pylint output status code
+
+result=0
+if [ $(($status&1)) -eq 1 ] || [ $(($status&2)) -eq 2 ] || [ $(($status&4)) -eq 4 ]
+then
+  result=1
+fi
+
+echo $result
diff --git a/scripts/compose_pki_core_packages b/scripts/compose_pki_core_packages
index 99d480c31a1b2c565055f24940c2b15355a79b31..af662792fe3675955413b3cd95894122b739d5f6 100755
--- a/scripts/compose_pki_core_packages
+++ b/scripts/compose_pki_core_packages
@@ -193,6 +193,22 @@ printf "#!/bin/bash\n\n"                 >  ${PKI_CORE_PACKAGE_SCRIPT}
 printf "${PKI_CORE_PACKAGE_COMMAND}\n\n" >> ${PKI_CORE_PACKAGE_SCRIPT}
 chmod 775 ${PKI_CORE_PACKAGE_SCRIPT}
 
+##
+## Run a pylint scan to check for warnings and errors in the python code
+##
+
+cd $PKI_DIR
+PYLINT_STATUS_CODE=`./pylint-build-scan.sh $PKI_PWD/$PKI_DIR`
+if [ $PYLINT_STATUS_CODE != 0 ]; then
+   echo
+   echo "###############################################################################"
+   echo "               !!!! Errors/Warnings reported by Pylint !!!!"
+   echo "###############################################################################"
+   echo -e "The complete report can be viewed at: \n    $PKI_PWD/$PKI_DIR/pylintscan/pylint-report.    "
+   echo -e "If the reported issues are false positives, please mark them ignored  \nin the configuration file '$PKI_PWD/$PKI_DIR/dogtag.pylintrc' \nand build again."
+   echo "###############################################################################"
+   exit 1
+fi
 
 ##
 ## Automatically invoke RPM/SRPM creation
-- 
1.7.10.2


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]