- 940 Views
- 2 replies
- 0 kudos
Resolved! Why this notebook is returning an error only when called by another notebook?
When I uncomment the last two lines of Called_Notebook.py and run it manually by itself, it correctly returns the output as:Status: SUCCESSCircle area: 50.26544But when I comment out the last two lines of Called_Notebook.py and run it from the Caller...
- 940 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @pradeep_singh, your suggestion worked. Thank you for sharing your knowledge. Worth noticing that not including dbutils.notebook.exit(f"{Value to return}") raised the error in the exception block of the function inside the Called_Notebook - and th...
- 0 kudos
- 1007 Views
- 3 replies
- 0 kudos
Getting driver error for my job when migrating to Unity
I am in the process of migrating our jobs from the legacy hive metastore to Unity. I have modified my existing job to read and write from a different bucket as part of the migration. The only change I have made to my job config is to enable this sett...
- 1007 Views
- 3 replies
- 0 kudos
- 569 Views
- 1 replies
- 1 kudos
matching sas proc survey means for quantiles in databricks
Hi, I currently have the following code in databricks that i am using to calculate survey estimates and quantiles. I wish to match (or get as close) to SAS results using proc survey means for quantiles as possible (I am able to match proportions fine...
- 569 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Jake3, I didn’t expect to jump into this thread given my complete lack of SAS knowledge and the rather serious-looking statistics you’re working with. But I decided to treat your post as a chance to see what Genie Code could do with it... and it...
- 1 kudos
- 1957 Views
- 1 replies
- 1 kudos
Resolved! Azure Databricks S3 External Location
Hi,I have recently created a new Azure Databricks account and several workspaces. I am needing to ingest data from an S3 bucket and am trying to follow the documentation detailed here:https://learn.microsoft.com/en-us/azure/databricks/connect/unity-c...
- 1957 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tsmith-11, Having checked internally and from the screenshot, this doesn’t look like a configuration issue on your side but rather that the cross‑cloud S3 feature isn’t enabled on your Azure Databricks account/metastore yet. You should see an AWS...
- 1 kudos
- 792 Views
- 3 replies
- 1 kudos
Refresh streaming table error
Refresh streaming table sql succeeds for the first time.The subsequent refresh statements fails with TABLE_OR_VIEW_NOT_FOUND error.The streaming table is still available in the same catalog and schema
- 792 Views
- 3 replies
- 1 kudos
- 1 kudos
HiHere is the query i use REFRESH STREAMING TABLE `cxxxxx`.`tgt_dev`.`ldp_csv` It is successful when i execute the first time.If i run the same query after 30 min, it throws error TABLE_OR_VIEW_NOT_FOUND
- 1 kudos
- 1119 Views
- 2 replies
- 0 kudos
Resolved! Creating a sync table from a workspace catalog to a project
We have a table in a workspace we would like to sync to a project. And we can fine choose the database project but we cannot see the database in the first section (Destination), see attached file.
- 1119 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Sega2, Thanks for sharing the screenshot — this helps clarify what's happening. There are two likely reasons why you're not seeing your database in the Destination section: 1. The source table must be in Unity Catalog Synced tables only support ...
- 0 kudos
- 8777 Views
- 7 replies
- 4 kudos
Databricks SQL Warehouse, Tableau and spark.driver.maxResultSize error
I'm attempting to create a tableau extract on tableau server with a connection to databricks large sql warehouse. The extract process fails due to spark.driver.maxResultSize error.Using a databricks interactive cluster in the data science & engineer...
- 8777 Views
- 7 replies
- 4 kudos
- 4 kudos
This is a common headache, I've definitely hit this wall myself when dealing with large datasets! My go-to is usually optimizing queries to return less data initially, maybe aggregating more in SQL. Have you considered checking your click speed on a ...
- 4 kudos
- 7091 Views
- 9 replies
- 4 kudos
Resolved! Directories added to the Python sys.path do not always work fine on executors for shared access mod
Let's assume we have a workspace folder containing two Python files.module1 with a simple addition function:def add_numbers(a, b): return a + bmodule2 with a dummy PySpark custom data source:from pyspark.sql.datasource import DataSource, DataSource...
- 7091 Views
- 9 replies
- 4 kudos
- 4 kudos
Hi all,Thanks for the feedback and proposed ideas.@szymon_dybczak Your idea of relative imports work when the module is hosted in a child directory to the current running notebook. It does not work if we need to go up one or two directories and navi...
- 4 kudos
- 2540 Views
- 5 replies
- 3 kudos
Resolved! Getting "socket closed" with query federation to Oracle DB on Amazon RDS
I am following this guide to connect to an Oracle DB in Amazon RDS: https://docs.databricks.com/aws/en/query-federation/oracle. I've created the connection, but when I go to test it, it loads for a while and then says "socket closed". From my underst...
- 2540 Views
- 5 replies
- 3 kudos
- 3 kudos
Accepting as answer because it was related to firewall blocking port 1521. After eliminating all the other options we dug deeper on the networking side and found the issue. Thanks!
- 3 kudos
- 1100 Views
- 1 replies
- 1 kudos
Resolved! Use .R file in data pipeline
In general R pipeline, we can use source("abc.R") file. However, it does not work in Databricks. I got errors: source("./abc.R"), but the error say: Error in file(filename, "r", encoding = encoding) : cannot open the connectionHow best to build a pip...
- 1100 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @NW1000 , I did some digging and have some helpful hints to share. The behavior you are seeing is a bit unintuitive if you’re coming from a local R setup. Here’s what’s going on. In Databricks, source("./abc.R") fails because the R process ...
- 1 kudos
- 2888 Views
- 5 replies
- 0 kudos
Reading JSON file to columns as relational ?
Reading JSON file to columns as relational ?====================================== Thanks for reviewing my threads. I like to explore Reading JSON file to columns as relational within Databricks.I have input file at workspace path > path\receipt.js...
- 2888 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @RIDBX, Thanks for the thorough description. Flattening JSON into relational columns is one of the most common data engineering tasks in Databricks, and there are several powerful approaches depending on your JSON structure. Let me walk you throug...
- 0 kudos
- 2596 Views
- 8 replies
- 3 kudos
Resolved! schema evolution with structured streaming: upstream schema change causes downstream writer fails.
Hello,Bronze: use classic or job compute, Autoloader with.option("mergeSchema", "true"). Schema evolution works correctly. data goes to bronze.my_bronze_table.Silver: uses serverless compute, reader reads bronze.my_bronze_table, does all necessary t...
- 2596 Views
- 8 replies
- 3 kudos
- 3 kudos
Hi @cdn_yyz_yul, Because the silver stream runs on serverless, you can’t relax state-store schema checks or set custom Spark configs. When the upstream bronze table schema evolves in a way that changes the schema of any stateful operator, the streami...
- 3 kudos
- 10252 Views
- 8 replies
- 3 kudos
Resolved! Multiple versions of custom libraries on the cluster
Using the install_libraries API, I installed a custom Python whl file on a running cluster. For certain types of requests, we have a requirement to install a different version of the same custom whl file in the running cluster. My problem is that uni...
- 10252 Views
- 8 replies
- 3 kudos
- 3 kudos
Why does databricks forum not filter out these spambot AI replies ? If one wants an AI chatbot to guess an answer their question, they are free to directly ask ChatGPT, etc. A community forum is for human-written answers based on actual first-hand ...
- 3 kudos
- 1016 Views
- 2 replies
- 2 kudos
Resolved! Seeking Best Approach for Bulk Migration of LUA/Exasol Scripts to Databricks PySpark
Hi All,We are planning a bulk migration of LUA Script / Exasol scripts to Databricks native PySpark and are evaluating the best approach for large-scale automated code conversion and testing in Databricks.So far, we have analyzed the following option...
- 1016 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Phani1, After some research, I don't believe there’s a Databricks-native, one-click tool to bulk-convert Lua/Exasol to PySpark. Databricks AI Assistant is great for interactive refactoring, but as you said, it’s not really a bulk‑migration engine...
- 2 kudos
- 1503 Views
- 4 replies
- 5 kudos
Resolved! optimizing my databricks code
I have the following code in databricks under serverless and i want to know how to improve it to make it more efficient and run faster without having the results change (standard errors change slightly when i try to improve it): # Databricks Serverle...
- 1503 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Jake3, Your Taylor-linearisation row-percent estimator is well structured. The main performance bottleneck is the Python-level loop over every domain/measure combination, with a full DataFrame copy (df.copy()) happening inside each iteration. Her...
- 5 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
