Using 1.3 Getting Started with the Databricks Platform Lab. to self learning. When I run DE 2.1 to setup environment, got following error:Configuration dbacademy.library.version is not available.Following is the code in the common setup.specified_ve...
Hello,I'm creating a DLT pipeline where I read a Kafka stream, perform transformations using UDFs, and save the data in multiple tables. When I define the functions directly in the same notebook, the code works fine. However, if I move the code into ...
Hi @YS1 ,Have you added the python file in the Pipeline settings, in the list of source code?
A delta live table pipeline reads a delta table on databricks. Is it possible to limit the size of microbatch during data transformation?I am thinking about a solution used by spark structured streaming that enables control of batch size using:.optio...
One other thought -- if you are considering using pandas_udf api, there is a way to control batch size there:pandas_udf guide note the comments there about arrow batch size params.
In databricks I can set a config variable at session level, but it is not found in the context variables:spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore spark.conf.get(f"dataset.bookstore")#123 scf = spark.sparkContext.getConf() allc =...
A while back I think I found a way to get python to list all the config values. I was not able to re-create it. Just make one of your notebook code sections scala (first line) and use the second line: %scala(spark.conf.getAll).foreach(println)
I found this old post (https://community.databricks.com/t5/data-engineering/databricks-cluster-web-terminal-different-permissions-with-tmux/td-p/26461) that was never really answered.I am having the same problem. If I am in the raw terminal, I can a...
Hi @Twilight, To resolve this, ensure the `tmux` session runs under the same user context as the raw terminal, verify environment variables are set correctly, initialize `tmux` with the same shell and environment settings, check for any ACLs on the `...
If I have a table which is clustered by (a, b, c) and I issue a query filtering on (b, c), will the query benefit from the optimization by the cluster of (a, b, c)?
Hi @oripsk, When you query a table clustered by columns (a, b, c) and filter on (b, c), the query will not fully benefit from the clustering optimization. Clustering works best when the query filter includes the leading column(s) in the clustering or...
On newer Databricks Runtime versions, %conda commands are not supported. You can use %pip commands instead:%pip list I have tested this on Databricks Runtime 15.4 LTS Beta.
Hello,I have tables created on unity catalog that point to the raw area , from these tables I need to create a data model (facts and dimensions) that will aggregate this data, transform certain things. Then I need to store in the Azure Datalake in de...
Hi @ad_k, To create a data model from Unity Catalog tables and store it in Azure data lake in Delta format, use Databricks Notebooks with PySpark or SQL. The process involves reading raw data from Unity Catalog, transforming it into fact and dimensio...
My SQL notebooks in databricks suffer from having at least several different counts of spaces between tab marks. It makes it very difficult to maintain pretty code spacing. What sets the tab spacing in SQL language notebooks, and how is it set/adju...
Hi @KFries, It sounds like you're dealing with a common issue with tab spacing in Databricks SQL notebooks. Databricks uses 2 spaces per tab by default, and while you can't change this setting globally, you can manually format your notebook via the "...
Hello AllI am new to Azure Data Bricks and trying to show the Azure data bricks table data onto UI using react JS. Lets say there 2 tables Emplyee and Salary , I need to join these two tables with empid and generate JSON out put and calling API (end ...
I am trying to use autoloader to load data from two different blobs from within the same account so that spark will discover the data asynchronously. However, when I try this, it doesn't work and I get the error outlined below. Can anyone point out w...
If were were to upgrade to ADLSg2, but retain the same structure, would there be scope for this method above to be improved (besides moving to notification mode)?
I'm trying to switch checkpoint locations from dbfs to abfss and i have noticed the following behaviour.The spark.sparkContext.setCheckpointDir will fail unless I call...dbutils.fs.mkdirs(checkpoint_dir) in the same cell.On top of this, the df = df....
i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'
Hiyou can try: my_df = spark.read.format("csv") .option("inferSchema","true") # to get the types from your data .option("sep",",") # if your file is using "," as separator .option("header","true") # if you...
Hello all,I am saving my data frame as a Delta Table to S3 and AWS Glue using pyspark and `saveAsTable`, so far I can do this but something curious happens when I try to change the `path` (as an option or as an argument of `saveAsTable`).The location...
Hi @mexcram, When saving a DataFrame as a Delta Table to S3 and AWS Glue using PySpark's `saveAsTable`, changing the `path` option or argument often results in the Glue table location being set to a placeholder path (e.g., `s3://my-bucket/my_table-__...
I am trying to debug my python script using Databricks VSCode extension. I am using udf and pandas_udf in my script. Everything works fine except when the execution gets to the udf and pandas_udf usages. It then complains that "SparkContext or SparkS...
Hi @SeyedA, To resolve this, first, ensure your SparkSession is properly initialized in your script. Be aware of the limitations of Databricks Connect, which might affect UDFs, and consider running UDFs locally in a simple Spark environment for debug...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group