- 14034 Views
- 11 replies
- 6 kudos
Resolved! Autocomplete parentheses, quotation marks, brackets and square stopped working
Hello guys, can someone help me?Autocomplete parentheses, quotation marks, brackets and square stopped working in python notebooks.How can I fix this?Daniel
- 14034 Views
- 11 replies
- 6 kudos
- 6 kudos
@Piper Wilson​ , @Werner Stinckens​ Thank you so much for your help.I made the suggestion of the @Jose Gonzalez​ and now it works.
- 6 kudos
- 3704 Views
- 2 replies
- 4 kudos
Resolved! Generating Spark SQL query using Python
I have a Spark SQL notebook on DB where I have a sql query likeSELECT * FROM table_name WHERE condition_1 = 'fname' OR condition_1 = 'lname' OR condition_1 = 'mname' AND condition_2 = 'apple' AND condition_3 ='orange'There are a lot ...
- 3704 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @John Constantine​ ,I think you can also use arrays_overlap() for your OR statements docs here
- 4 kudos
- 3559 Views
- 4 replies
- 5 kudos
Resolved! Conditionally create a dataframe
I would like to implement a simple logic:if Df1 is empty return Df2 else newDf = Df1.union(Df2)May happened that Df1 is empty and the output is simply: []. In that case I do not need union.I have it like this but getting error when creating datafra...
- 3559 Views
- 4 replies
- 5 kudos
- 5367 Views
- 2 replies
- 1 kudos
- 5367 Views
- 2 replies
- 1 kudos
- 1 kudos
@Vaibhav Gour​ , It kinda depends on the case:if there are no workers available when your job starts, you get an error. As the cluster is unable to start so code cannot be executed. But this is not an autoscale issue.If you need to scale up, but for ...
- 1 kudos
- 10877 Views
- 6 replies
- 4 kudos
Resolved! Object of type bool_ is not JSON serializable
I am doing a convertion of a data frame to nested dict/json. One of the column called "Problematic__c" is boolean type.For some reason json does not accept this data type retriving error: "Object of type bool_ is not JSON serializable" I need this as...
- 10877 Views
- 6 replies
- 4 kudos
- 11518 Views
- 4 replies
- 8 kudos
Resolved! Is there a way to submit multiple queries to data bricks SQL END POINT using REST API ?
Is there a way to submit multiple queries to data bricks SQL END POINT using REST API ?
- 11518 Views
- 4 replies
- 8 kudos
- 8 kudos
@Manoj Kumar Rayalla​ DBSQL currently limits execution to 10 concurrent queries per cluster so there could be some queuing with 30 concurrent queries. You may want to turn on multi-cluster load balancing to horizontally scale with 1 more cluster for...
- 8 kudos
- 6939 Views
- 0 replies
- 0 kudos
ColumnTransformer not fitted after sklearn Pipeline loaded from Mlflow
I am building a machine learning model using sklearn Pipeline which includes a ColumnTransformer as a preprocessor before the actual model. Below is the code how the pipeline is created.transformers = [] num_pipe = Pipeline(steps=[ ('imputer', Si...
- 6939 Views
- 0 replies
- 0 kudos
- 2999 Views
- 3 replies
- 3 kudos
Is there an alerting API please?
Is there an alerting api so that alerts can be source controlled and automated, please ?https://docs.databricks.com/sql/user/alerts/index.html
- 2999 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @Nick Hughes​ , as of today we do not expose or document the API for these features. I think it will be a useful feature so I created an internal feature request for it (DB-I-4289). If you (or any future readers) want more information on this f...
- 3 kudos
- 4581 Views
- 6 replies
- 2 kudos
How not to reprocess old files without delta ?
Hi guys,​Look that case: Company ACME (hypothetical company)​This company does not use delta, but uses open source Spark to process raw data for .parquet, we have a 'sales' process which consists of receiving every hour a new dataset (.csv) within th...
- 4581 Views
- 6 replies
- 2 kudos
- 2 kudos
Hi @Jose Gonzalez​ , ​I agree the best option is to use auto load, but some cases you don`t have the databricks plataform and don`t use delta, i this cases you need build a way to process the new raw files
- 2 kudos
- 10228 Views
- 5 replies
- 0 kudos
How to filter files in Databricks Autoloader stream
I want to set up an S3 stream using Databricks Auto Loader. I have managed to set up the stream, but my S3 bucket contains different type of JSON files. I want to filter them out, preferably in the stream itself rather than using a filter operation.A...
- 10228 Views
- 5 replies
- 0 kudos
- 0 kudos
According to the docs you linked, the glob filter on input-path only works on directories, not on the files themselves.So if you want to filter on certain files in the concerning dirs, you can include an additional filter through the pathGlobFilter o...
- 0 kudos
- 17272 Views
- 6 replies
- 3 kudos
I want to use databricks workers to run a function in parallel on the worker nodes
I have a function making api calls. I want to run this function in parallel so I can use the workers in databricks clusters to run it in parallel. I have tried with ThreadPoolExecutor() as executor: results = executor.map(getspeeddata, alist)to run m...
- 17272 Views
- 6 replies
- 3 kudos
- 3 kudos
You guys are not getting the point, I am making API calls in a function and want to store the results in a dataframe. I want multiple processes to run this task in parallel. How do I create a UDF and use it in a dataframe when the task is calling an ...
- 3 kudos
- 19721 Views
- 2 replies
- 1 kudos
ModuleNotFoundError / SerializationError when executing over databricks-connect
I am running into the following error when I run a model fitting process over databricks-connect.It looks like worker nodes are unable to access modules from the project's parent directory. Note that the program runs successfully up to this point; n...
- 19721 Views
- 2 replies
- 1 kudos
- 1 kudos
@Sarosh Ahmad​ , Could you try adding the zip of the module to the addPyFile like belowspark.sparkContext.addPyFile("src.zip")
- 1 kudos
- 834 Views
- 0 replies
- 0 kudos
Hi juliet Wu I have completed my databricks apache spark associate developer exam on 7/10/2021 after subsequent completion of my exam I got my badge t...
Hi juliet Wu I have completed my databricks apache spark associate developer exam on 7/10/2021 after subsequent completion of my exam I got my badge to my Webaccesor Mail immediately after 1 day of exam which is on 8/10/2021​.but I didn't received my...
- 834 Views
- 0 replies
- 0 kudos
- 3167 Views
- 1 replies
- 2 kudos
Resolved! How to source control a Dashboard?
Is it possible to source control the dashboard along with a notebook code? When source controlling a python notebook it gets converted to *.py. It looks like the resulting *.py file loses the information about the dashboard cells. Thus, if this *.py ...
- 3167 Views
- 1 replies
- 2 kudos
- 2 kudos
No, you will need to save as another source, like DBC Archive, to replicate the Notebook features.
- 2 kudos
- 4704 Views
- 1 replies
- 1 kudos
How to provide custom class extending SparkPlugin/ExecutorPlugin in Databricks 7.3?
How to properly configure the jar containing the class and spark plugin in Databricks?During DBR 7.3 cluster creation, I tried by setting the spark.plugins, spark.driver.extraClassPath and spark.executor.extraClassPath Spark configs by copying the ja...
- 4704 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Krishna Kashiv​ - I don't know if we've met yet. My name is Piper and I'm a community moderator here. Thank you for your new question. It looks thorough! Let's give it a while to see what our members have to say. Otherwise, we will circle back...
- 1 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
