- 1908 Views
- 2 replies
- 0 kudos
Autoloader with file notifications on a queue that is on a different storage account to the blobs
Hello,I am trying to set up autoloader using file notifications but as the storage account we are reading from is a premium storage account we have setup event subscriptions to pump the blob events to queues that exist on a standard gen 2 storage acc...
- 1908 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @ChrisLawford_n1 So in your case, it's not able to resolve the file paths from the event notificationsbecause they're pointing to a different storage account (Storage Account 1), which is not associated with the queue.Use a StorageV2 Account for B...
- 0 kudos
- 796 Views
- 2 replies
- 0 kudos
Unable to fetch 3rd level array using dot notation from a json data in local spark via VS Code
I am able to fetch 3rd level array using dot notation from a json in Databricks but same code is not working in local spark via VS Code.Example - df.select(F.col('x.y.z')) where x is array, y is array and z is also an array.In local spark I am gettin...
- 796 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ritasree ,When you're saying about local spark you mean that you've configured it locally, or are you using Databricks Connect? If you're configured spark locally, then check what version of Spark you're using.
- 0 kudos
- 1533 Views
- 2 replies
- 0 kudos
Resolved! Legacy hive_metatstore corruption
I am seeing some legacy hive_metastore corruption (especially tables created as parquet instead of Delta) lately in my client's place, who is in the midst of migrating to UC. We were provided with a Scala code to remove the erroneous Parquet files ph...
- 1533 Views
- 2 replies
- 0 kudos
- 0 kudos
HI @thbeh_com Yes, this is a fairly common issue during UC migrations, especially with legacy Hive metastore tables. The corruption typically happens because:- Metadata-data misalignment - Hive metastore references files that no longer exist or have ...
- 0 kudos
- 3743 Views
- 1 replies
- 1 kudos
Resolved! Spark Structured Streaming Timeout Waiting for KafkaAdminClient Node Assignment on Amazon MSK
Hello! I’m having trouble establishing a Kafka connection between my Databricks notebook and my Kafka server in Amazon MSK. I’ve run some tests and I’m really stuck—I hope someone can help me.I have two brokers. First, I checked connectivity with:%sh...
- 3743 Views
- 1 replies
- 1 kudos
- 1 kudos
We found the solution!, thanks to a Databricks architect here’s how we ultimately fixed it: 1.- Copy the JVM’s cacerts into a volume so Spark can trust Amazon’s MSK certificate bundle. From a notebook cell with shell access, run:%sh JAVA_HOME=$(dirna...
- 1 kudos
- 1125 Views
- 1 replies
- 1 kudos
Resolved! How Auto Loader works – file level or row level?
Does Auto Loader work on file level or row level? If it works on file level and does not process the same file again, then how can we make it pick only the new rows when data is appended to that file?
- 1125 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Akshay_Petkar ,Autoloader works on file level. Now, by default autoloader is configured with following option:cloudFiles.allowOverwrites = falseSo, above option causes files to be processed exactly once. But when you switch this option to true, t...
- 1 kudos
- 6144 Views
- 8 replies
- 1 kudos
Resolved! Accessing parameter defined in python notebook into sql notebook.
Hi All,I have one python notebook(../../config/param_notebook), where all parameters are defined, like:dbutils.widgets.text( "catalog", "catalog_de")spark.conf.set( "catalog.name", dbutils.widgets.get( "catalog"))dbutils.widgets.text( "schema", "emp"...
- 6144 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi all,I have a SQL notebook that contains the following statement:CREATE OR REPLACE MATERIALIZED VIEW ${catalog_name}.${schema_name}.emp_table ASSELECT ...I’ve configured the values for catalog_name and schema_name as pipeline parameters in my DLT p...
- 1 kudos
- 1160 Views
- 1 replies
- 1 kudos
Resolved! Notebook dashboard export unavailable
Hello,Recent changes in the databricks notebook dashboards have removed the option to download the dashboard as HTML.Previously it was possible to download it from the notebook dashboard view. Currently it's only possible to download the notebook its...
- 1160 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @amrim! You're right to flag this, thank you for bringing it up. I’ll check internally for any upcoming changes regarding this feature or alternative ways to download the notebook dashboard in HTML format. I’ll get back to you once I have an up...
- 1 kudos
- 1263 Views
- 1 replies
- 0 kudos
Reading a large zip file containing NDJson file in Databricks
Hi,We have a 5 GB ZIP file stored in ADLS. When uncompressed, it expands to approximately 115 GB and contains multiple NDJSON files, each around 200 MB in size. We need to read this data and write it to a Delta table in Databricks on a weekly basis.W...
- 1263 Views
- 1 replies
- 0 kudos
- 0 kudos
Unzip the Archive FileApache Spark cannot directly read compressed ZIP archives, so the first step is to decompress the 5 GB file. Since the uncompressed size is substantial (115 GB), the process must be handled carefully to avoid overwhelming the dr...
- 0 kudos
- 3538 Views
- 5 replies
- 3 kudos
Resolved! Enable to use library GraphFrames
Hello,I am trying to install and use the library GraphFrames but keep receiving the following error: "AttributeError: 'SparkSession' object has no attribute '_sc'".I have tried to install the library on my all-purpose cluster (Access mode: Standard)....
- 3538 Views
- 5 replies
- 3 kudos
- 3 kudos
@szymon_dybczak Thanks for the responses. I indeed changed my all-purpose cluster access mode and it worked. I figured that was a nicest option than changing the runtime.
- 3 kudos
- 4399 Views
- 2 replies
- 0 kudos
Resolved! Use of Python variable in SQL cell
If using spark.conf.set(<variable_name>, <variable_value>), or just referring a widget value directly, in a Python cell and then referring to it in a SQL cell with ${variable_name} one gets the warning: "SQL query contains a dollar sign parameter, $p...
- 4399 Views
- 2 replies
- 0 kudos
- 4936 Views
- 1 replies
- 0 kudos
Oracle JDBC Load Fails with Timestamp Partitioning (lowerBound/upperBound)
Hi everyone,I'm trying to read data from an Oracle database into Databricks using JDBC with timestamp-based partitioning. However, it seems that the partitioning doesn't work as expected when I specify lowerBound and upperBound using timestamp string...
- 4936 Views
- 1 replies
- 0 kudos
- 0 kudos
@pavlosskev Could you try adding the following option as well to your read? .option("sessionInitStatement", "ALTER SESSION SET NLS_TIMESTAMP_FORMAT = 'YYYY-MM-DD HH24:MI:SS'") df = ( spark.read.format("jdbc") .option("url", jdbcUrl) .opti...
- 0 kudos
- 1581 Views
- 1 replies
- 1 kudos
Resolved! E series vs F series VM's
Hi all,I need to run weekly maintenance on approximately 7,000 tables in my Databricks environment, involving OPTIMIZE, VACUUM, and ANALYZE TABLE (for statistics calculation) on all tables.My question is: between the Ev4, Edv4, and Fsv2 VM series, wh...
- 1581 Views
- 1 replies
- 1 kudos
- 1 kudos
@Sainath368 OPTIMIZE and VACUUM are compute-intensive operations, so you can choose a compute-optimized instance like the F series for both drivers and workers, which has a higher CPU-to-memory ratio. If its UC managed table, I recommend enabling Pr...
- 1 kudos
- 33264 Views
- 4 replies
- 4 kudos
Resolved! PySpark: Writing Parquet Files to the Azure Blob Storage Container
Currently I am having some issues with the writing of the parquet file in the Storage Container. I do have the codes running but whenever the dataframe writer puts the parquet to the blob storage instead of the parquet file type, it is created as a f...
- 33264 Views
- 4 replies
- 4 kudos
- 4 kudos
This is my approach:from databricks.sdk.runtime import dbutils from pyspark.sql.types import DataFrame output_base_url = "abfss://..." def write_single_parquet_file(df: DataFrame, filename: str): print(f"Writing '{filename}.parquet' to ABFS") ...
- 4 kudos
- 1355 Views
- 1 replies
- 0 kudos
How to create a SparkSession in jobs run-unit-tests
I’m converting my Python unit tests to run with databricks jobs run-unit-tests.Each test needs a SparkSession, but every pattern I try What I tried1. Create my own local Sparkspark = (SparkSession.builder.master("local[*]").appName("unit-test").getOr...
- 1355 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @yhu126 ,Maybe below blog post give you some inspiration:Writing Unit Tests for PySpark in Databricks: Appr... - Databricks Community - 122398
- 0 kudos
- 5068 Views
- 7 replies
- 1 kudos
Resolved! Unable to resolve column error while trying to query the view
I have a federated table from snowflake in data bricks say employee.When i executed print schema i am able to see schema as "employeeid": long,"employeename":stringTried to create a view as create view vw_emp with schema binding as select `"employeei...
- 5068 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @nkrom456 ,Try something like this. If you are using backticks it treats a column name exactly as you type (in this case it treats double quotes as a part of a colum name)create view vw_emp with schema binding as select `employeeid` from employee ...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 577 | |
| 349 | |
| 287 |
