- 3149 Views
- 2 replies
- 4 kudos
Resolved! Connecting Delta Tables from any Tools
Hi Team,To access SQL Tables we use tools like TOAD , SQL SERVER MANAGEMENT STUDIO (SSMS).Is there any tool to connect and access Databricks Delta tables.Please let us know.Thank you
- 3149 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Rajesh Vinukonda​ Hope you are doing well. Thanks for sending in your question. Were you able to find a solution to your query?
- 4 kudos
- 27510 Views
- 7 replies
- 7 kudos
Resolved! Performance for pyspark dataframe is very slow after using a @pandas_udf
Hello,I am currently working on a time series forecasting with FBProphet. Since I have data with many time series groups (~3000) I use a @pandas_udf to parallelize the training. @pandas_udf(schema, PandasUDFType.GROUPED_MAP) def forecast_netprofit(pr...
- 27510 Views
- 7 replies
- 7 kudos
- 7 kudos
Thank you for the answers. Unfortunately this did not solve the performance issue.What I did now is I saved the results into a table:results.write.mode("overwrite").saveAsTable("db.results") This is probably not the best solution but after I do that ...
- 7 kudos
- 1529 Views
- 2 replies
- 2 kudos
Resolved! java.lang.NoSuchMethodError in databricks
I have created a package. Now I am calling a method from this package in my notebook but it is throwing me java.lang.NoSuchMethodError in databricks. The method exists in the package. Can you please guide me regarding the same.Thanks!
- 1529 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi! I am sharing the error stack with you. I can't share the code with you due to confidentiality of the code. Can you please guide me ?java.lang.NoSuchMethodError: com.iig.utils.common.IIGCommonConstants$.flowProperties()Ljava/lang/String; at com.ii...
- 2 kudos
- 4129 Views
- 8 replies
- 3 kudos
Resolved! How to force pandas_on_spark plots to use all dataframe data?
When I load a table as a `pandas_on_spark` dataframe, and try to e.g. scatterplot two columns, what I obtain is a subset of the desired points. For example, if I try to plot two columns from a table with 1000000 rows, I only see some of the data - i...
- 4129 Views
- 8 replies
- 3 kudos
- 3 kudos
Hi @Davide Cagnoni​ , The Ideas Portal lets you influence the Databricks product roadmap by providing feedback directly to the product team. Use the Ideas Portal to:Enter feature requests.View, comment, and vote up other users’ requests.Monitor the p...
- 3 kudos
- 4354 Views
- 4 replies
- 4 kudos
Resolved! Bronze silver gold layers
Is there a best practise guide on setting up the delta lake for these 3 layers. ​I'm looking for document or scripts to run that will assist me.
- 4354 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Lloyd Vickery​ , Did you find any of the above answers helpful? Would you like to tell us if you solved it?
- 4 kudos
- 2705 Views
- 5 replies
- 1 kudos
Resolved! Convert table in nested JSON
What is the easiest way to convert a table to a nested JSON?
- 2705 Views
- 5 replies
- 1 kudos
- 1 kudos
@Sergio Paz​ - How's it going? Are you able to give us more information?
- 1 kudos
- 3988 Views
- 3 replies
- 2 kudos
Resolved! Databricks runtime from docker hub image
Hello,are databricks runtimes from docker hub ( https://hub.docker.com/r/databricksruntime/standard ) same as actual runtimes inside Databricks? I mean when we made our own docker image from databricksruntime/standard will be there same dependencies...
- 3988 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @michael henzl​ ,Just checking if you still need help with this or not any more? please let us know
- 2 kudos
- 2343 Views
- 2 replies
- 2 kudos
Resolved! create a dataframe with all the responses from the api requests within foreachPartition
I am trying to execute an api call to get an object(json) from amazon s3 and I am using foreachPartition to execute multiple calls in paralleldf.rdd.foreachPartition(partition => { //Initialize list buffer var buffer_accounts1 = new ListBuffer[St...
- 2343 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Sandesh Puligundla​ ,Thank you for sharing the solution. We will mark it as "best" response so, in the future is another user has the same question, they will be able to find the solution right away.
- 2 kudos
- 1356 Views
- 2 replies
- 3 kudos
Resolved! Can't view files of different types in databricks
I am reading a Kafka input using Spark Streaming on databricks and trying to deserialize it. The input is in the form of thrift. I want to create a file of .thrift format to provide schema but am unable to do it. Even if I create the file locally and...
- 1356 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @John Constantine​ ,Just checking if you still need help or not anymore. If you do, please share as much details and logs as possible, so we would be able to help better.
- 3 kudos
- 1570 Views
- 3 replies
- 7 kudos
Resolved! ETL in Databricks
I use Azure Databricks for ETL. I read/write data from and to raw/stage/curate folders. I write dataframe to a path (eg: /mnt/datalake/curated/....). In final step I read data from the path, convert that to dataframe and write it to the Azure SQL DB/...
- 1570 Views
- 3 replies
- 7 kudos
- 7 kudos
Hi @Kris Koirala​ ,Just checking if you still have any follow-up questions? please let us know.
- 7 kudos
- 3010 Views
- 4 replies
- 1 kudos
Resolved! Method iterableAsScalaIterable does not exist Pydeequ
Hello,I'm using Databricks and pydeequ to build a QA step in structured streaming.One of the Analyzers that I need to use is the Uniqueness.If I try to add another one like Completeness, work properly, but if y add the Uniqueness I get an error:py4j....
- 3010 Views
- 4 replies
- 1 kudos
- 1 kudos
I think it is because you did not attach the libraries to the cluster.When you work with a notebook, the sparksession is already created.To add libraries, you should install them on the cluster (in the compute tab) using f.e. pypi/maven etc.
- 1 kudos
- 2832 Views
- 4 replies
- 0 kudos
Resolved! Databricks Cluster create fail
i facing the problem here in creating cluster in databricks. Error as below :MessageCluster terminated.Reason:Unexpected launch failureAn unexpected error was encountered while setting up the cluster. Please retry and contact Databricks if the proble...
- 2832 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Giin Sing Wong​ ,Just a friendly follow-up. Is this issue still happening or you were able to resolve it by increasing your account's quota? Please let us know.
- 0 kudos
- 49622 Views
- 8 replies
- 2 kudos
Resolved! Run stored bash in Databricks with %sh
Hi, I made bash file in databricks and I can see that the file is stored as the following picture. I was supposed to run this bash file through %sh cell, but as you see the following picture, I could not find bash file, which I could find through d...
- 49622 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @keunsoop​ ,Are you able to run your code using an init script? I would like to share some docs in case you might have some questions https://docs.databricks.com/clusters/init-scripts.html
- 2 kudos
- 2918 Views
- 6 replies
- 5 kudos
Resolved! Databricks-Connect shows different partitions than Databricks for the same delta table
Hello,here is a small code-snippet:from pyspark.sql import SparkSession spark = SparkSession.builder.appName('example_app').getOrCreate() spark.sql('SHOW PARTITIONS database.table').show() The output inside the Databricks-Notebook:+-------------+--...
- 2918 Views
- 6 replies
- 5 kudos
- 2733 Views
- 2 replies
- 2 kudos
Resolved! [Simba][SparkJDBCDriver](500177) Error getting http path from connection string
I'm trying to use a very basic java program to connect to Databricks using spark jdbc driver (SparkJDBC42.jar), but I get the error (mentioned above): [Simba][SparkJDBCDriver](500177) Error getting http path from connection stringHere is my code snip...
- 2733 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Claude Repono​ Thank you for posting your question in the community. It seems you were able to find the solution by yourself. That's awesome. We are going to go ahead and mark your answer as the best solution.
- 2 kudos