- 3168 Views
- 1 replies
- 0 kudos
Error when running Spark-DL notebooks
I try several Spark Deep Learning inference notebooks on Windows. I run Spark in standalone mode with 1 worker with 12 cores (both driver-memory and executor-memory are set to 8G). I always get the same error when applying the deep learning model to ...
- 3168 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a connectivity issue. Check your connectivity by doing "%sh nc -zv {hostname} {port}" from your notebook
- 0 kudos
- 2413 Views
- 1 replies
- 0 kudos
Issue importing library deltalake
Hello,I'm currently seeing a rather cryptic error message whenever I try to import the deltalake library into Databricks (without actually doing anything else).import datalake"ImportError: /local_disk0/.ephemeral_nfs/envs/pythonEnv-cbe496f6-d064-40ae...
- 2413 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you trying to import this library to a Databricks notebook? are you using open source spark in your local machine?
- 0 kudos
- 2069 Views
- 1 replies
- 0 kudos
Getting "Job aborted" exception while saving data to the database
Hi!We have job, that runs every hour. It extracts data from the API and saves to the databricks table.Sometimes job fails with error "org.apache.spark.SparkException". Here is the full error:An error occurred while calling o7353.saveAsTable. : org.ap...
- 2069 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you have any NULL values in your data? Please verify that you data is valid
- 0 kudos
- 10914 Views
- 11 replies
- 3 kudos
Resolved! Replay(backfill) DLT CDC using kafka
Hello,We are receiving DB CDC binlogs through Kafka and synchronizing tables in OLAP system using the apply_changes function in Delta Live Table (DLT). A month ago, a column was added to our table, but due to a type mismatch, it's being stored incorr...
- 10914 Views
- 11 replies
- 3 kudos
- 3 kudos
Thank you @532664 for your detailed response! That's seems to me a very good solution, and it also helps me with my doubts
- 3 kudos
- 6261 Views
- 4 replies
- 0 kudos
Unable to read from azure blob using SAS token\\
I have tried many times all the answers from the internet and stackover flowI have already created the config section before this steps, it passed but this below step is not executing.
- 6261 Views
- 4 replies
- 0 kudos
- 0 kudos
We were getting this problem when using directory-scoped SAS tokens. While I know there are a number of potential issues that can cause this problem, one potential explanation is that it turns out there is an undocumented spark setting needed on the ...
- 0 kudos
- 3019 Views
- 0 replies
- 1 kudos
connect azure databricks to reverso using API
Hi Databricks community,Reverso is a language translation tool that converts texts from one language to another. I need to convert hundreds of text but its time consuming. Hence, I want to use its API to automate the process.How can I achieve this in...
- 3019 Views
- 0 replies
- 1 kudos
- 8224 Views
- 5 replies
- 0 kudos
What are optimized solutions for moving on-premise IBM DB2 CDC data to Databricks Delta table
Hi Team,My requirement is to move build a solution to move zos(db2) CDC data to Delta table on Realtime bases(at least near realtime) , data volume and number of tables are little huge (100 tables) I have researched I dont find any inbuild options in...
- 8224 Views
- 5 replies
- 0 kudos
- 3840 Views
- 1 replies
- 0 kudos
Resolved! Url column issue in UC information_schema.schemata view definition
Hi, I recently observed that, after creating a new catalog (without a managed location) in Unity Catalog, a column named 'url' is included in the definition of the information_schema.schemata view.However, there is no url column in the underlying tab...
- 3840 Views
- 1 replies
- 0 kudos
- 8147 Views
- 3 replies
- 0 kudos
Resolved! ignoreCorruptFiles behavior with CSV and COPY INTO
HiI'm using the COPY INTO command to insert new data (in form of CSVs) into an already existing table.The SQL query takes care of the conversion of the fields to the target table schema (well, there isn't other way to do that), and schema update is n...
- 8147 Views
- 3 replies
- 0 kudos
- 0 kudos
I actually found an option that could solve the newline issue I mentioned in my previous post:setting spark.sql.csv.parser.columnPruning.enabled to false withspark.conf.set("spark.sql.csv.parser.columnPruning.enabled", False)will consider malformed r...
- 0 kudos
- 4243 Views
- 1 replies
- 0 kudos
Best practice for Azure Key vault secrets in spark config
HelloI created a compute in which I refer the secret inside the spark config like this: spark.hadoop.fs.azure.account.key.xxxxxxxxxx.dfs.core.windows.net {{secrets/kv-xxxxxxx-xxxx/secret-name}} This, however, gives me the following warning. I've l...
- 4243 Views
- 1 replies
- 0 kudos
- 0 kudos
Extra info: I have used the format following the instructions on this page for spark configurationhttps://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage#:~:text=Use%20the%20following%20format%20to%20set%20the%20cluster%20Spa...
- 0 kudos
- 13234 Views
- 1 replies
- 0 kudos
Resolved! Problems with DLT, Unity catalog and external connection
I have following code:org = spark.read.table("catalog.dbo.organisation") @dlt.create_table() def organization(): return orgThe catalog is an external azure sql database (using external connector)When i validate this in Delta live table workflow I...
- 13234 Views
- 1 replies
- 0 kudos
- 0 kudos
use preview channel while create pipeline instead of current
- 0 kudos
- 3785 Views
- 2 replies
- 0 kudos
Unable to load data in DLT tables from Federated data sources
I tried to run this query and failing to load the data .What do I need to do load from federated data sources using DLT if this is not correct CREATE OR REPLACE LIVE TABLE bulkuploadhistory COMMENT 'Table generated for bulkuploadhistory.' TBLPROPERTI...
- 3785 Views
- 2 replies
- 0 kudos
- 0 kudos
Considering the databricks release cycles , is there any tenative dates that I can be aware of so that I can plan for my production releases
- 0 kudos
- 2802 Views
- 2 replies
- 0 kudos
Development Feedback Loop
I've noticed that the current development cycle for DLT jobs is quite time-consuming. The process of coding, saving, running in a workflow, and debugging seems arduous, and the feedback loop is slow. Is there a way to run DLT jobs without relying on ...
- 2802 Views
- 2 replies
- 0 kudos
- 12585 Views
- 2 replies
- 3 kudos
Resolved! How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?
val spark:SparkSession = SparkSession.builder() .master("local[3]") .appName("SparkByExamples.com") .getOrCreate()//Spark Read CSV Fileval df = spark.read.option("header",true).csv("address.csv")//Write DataFrame to address directorydf.write...
- 12585 Views
- 2 replies
- 3 kudos
- 3 kudos
Will your csv have the name prefix 'part-' or can you name it whatever you like?
- 3 kudos
- 8194 Views
- 6 replies
- 0 kudos
Unsupported datatype 'TimestampNTZType' with liquid clustering
I'm experimenting with liquid clustering and have some questions about compatible types (somewhat similar to Liquid clustering with boolean columns ).Table created as CREATE TABLE IF NOT EXISTS <TABLE> ( _time DOUBLE , timestamp TIMESTAMP_NT...
- 8194 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi,just educated guess:There is limitation in liquid clustering docs: You can only specify columns with statistics collected for clustering keysPerhaps it is related to data types for which you can collect statistics?But i could not find related docs...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
