- 901 Views
- 1 replies
- 0 kudos
Resolved! How to use GLOW in Databricks Premium on AWS?
Hi!Have connected workspace to AWS, but when I execute in a new notebook: %python %pip install glow.py import glow from pyspark.sql import SparkSession # Create a Spark session spark = (SparkSession.builder .appName("Genomics Analysis") ...
- 901 Views
- 1 replies
- 0 kudos
- 0 kudos
Solved this with the help of colleagues at last. First of all, it won't work with Serverless mode, so a cluster is required. Once the cluster is created in Compute section, on Library tab add those 2 libraries:Then running:import glow from pyspark.sq...
- 0 kudos
- 2088 Views
- 2 replies
- 0 kudos
Python callback functions fail to trigger
How can I get sys.exceptionhook and atexit module to trigger a callback function on exit of a python notebook? These fail to work when an unhandled exception is encountered (exceptionhook), or the program exits (atexit).
- 2088 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Lozik,Ran into this myself as well. The reason this doesn't work is because Databricks is using Ipython under the hood.The following codesnippet creates an exception hook for all exceptions (using the general Exception), it's also possible to spe...
- 0 kudos
- 1182 Views
- 1 replies
- 0 kudos
Databricks read CDF by partitions for better performance?
I’m working with a large dataframe in Databricks, processing it in a streaming-batch fashion (I’m reading as a stream, but using .trigger(availableNow=True) for batch-like processing).I’m fetching around 40GB of CDF updates daily and performing some ...
- 1182 Views
- 1 replies
- 0 kudos
- 0 kudos
You can indeed leverage your partitioned column to read and process Change Data Feed (CDF) changes in partitions. This approach can help you manage the processing load and improve performance. Here's a general outline of how you can achieve this:1. *...
- 0 kudos
- 2033 Views
- 2 replies
- 0 kudos
Handling Binary Files Larger than 2GB in Apache Spark
I'm trying to process large binary files (>2GB) in Apache Spark, but I'm running into the following error:File format is : .mf4 (Measurement Data Format) org.apache.spark.SparkException: The length of ... is 14749763360, which exceeds the max length ...
- 2033 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @pra18, You can split and load the binary files using split command like this. ret = os.system("split -b 4020000 -a 4 -d large_data.dat large_data.dat_split_")
- 0 kudos
- 1633 Views
- 2 replies
- 0 kudos
Databricks Workflow Orchestration
I have 50 tables and will increase gradually, so I want to create a single workflow to orchestrate the job and run it table-wise. Is there an option to do this in Databricks workflow?
- 1633 Views
- 2 replies
- 0 kudos
- 0 kudos
Breakup these 50 tables logically or functionally and place them in their own workflows. A good strategy would be to group tables that are dependent in the same workflow. Then use a master workflow to trigger each child workflow. So it will be like a...
- 0 kudos
- 1845 Views
- 1 replies
- 0 kudos
Dashboard
Hi I want to design a dashboard that will show some variables of Spark-UI. Is it possible to access Spark-UI variables from my spark program.
- 1845 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @subhas_1729, You can achieve this by leveraging Spark's monitoring and instrumentation APIs. Spark provides metrics that can be accessed through the SparkListener interface as well as the REST API. The SparkListener interface allows you to receiv...
- 0 kudos
- 1790 Views
- 3 replies
- 1 kudos
How to Apply row_num in DLT
Hi all,how to use row_num in DLT or What is the alternative for row_num function in DLT.We are looking for same functionality which row num is doing. Thanks in advance.
- 1790 Views
- 3 replies
- 1 kudos
- 1 kudos
@dbhavesh I apologize for the lack of explanation.The ROW_NUMBER function requires ordering over the entire dataset, making it a non-time-based window function. When applied to streaming data, it results in the "NON_TIME_WINDOW_NOT_SUPPORTED_IN_STREA...
- 1 kudos
- 2229 Views
- 6 replies
- 0 kudos
Unable to log in into Community Edition
So I just registered for the Databricks Community Edition. And received an email for verification.When I click the link, I'm redirected to this website (image attached) where I am asked to input email. And when I do that, it sends me a verification c...
- 2229 Views
- 6 replies
- 0 kudos
- 8364 Views
- 4 replies
- 1 kudos
- 8364 Views
- 4 replies
- 1 kudos
- 1822 Views
- 2 replies
- 0 kudos
Issue when connecting with Databricks cluster 15.4 without unity catalog using databricks connect
Hi,I have a shared cluster created on databricks which uses 15.4 runtime.I dont want to enable the unity catalog for this cluster.Previously I used python 3.9.13 version to connect to 11.3 cluster using databricks connect 11.3Now my company has restr...
- 1822 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @prasidataengine, For DBR runtime 13.3 LTS and above you must have Unity Catalog enabled to be able to use databricks-connect. A Databricks account and workspace that have Unity Catalog enabled. See Set up and manage Unity Catalog and Enable a wo...
- 0 kudos
- 1092 Views
- 2 replies
- 0 kudos
MongoDB Streaming Not Receiving Records in Databricks
Batch Read (spark.read.format("mongodb")) works fine.Streaming Read (spark.readStream.format("mongodb")) runs but receives no records.Batch Read (Works):df = spark.read.format("mongodb")\.option("database", database)\.option("spark.mongodb.read.conne...
- 1092 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @vidya_kothavale, MongoDB requires the use of change streams to enable streaming. Change streams allow applications to access real-time data changes without polling the database. Ensure that your MongoDB instance is configured to support change...
- 0 kudos
- 740 Views
- 1 replies
- 0 kudos
The lifetime of files in the DBFS are NOT tied to the lifetime of our cluster
What happen so that the lifetime of files in the DBFS are NOT tied to the lifetime of our cluster?
- 740 Views
- 1 replies
- 0 kudos
- 0 kudos
files in dbfs are typically not linked to a cluster or it's lifetime.There are tmp directories in dbfs so perhaps you are looking at those, but f.e. Filestore can definitely be used.However, I suggest not using dbfs but some data lake (S3/ADLS).
- 0 kudos
- 1047 Views
- 1 replies
- 0 kudos
Unable to run selected lines in Databricks
I'm using SQL language in databricks. Basically I'm a tester and I'm trying to test the data load on tables by writing various queries. I'm unable to select a particular query and run. Ctrl+Shift+Enter shortcut is not working.Currently I need to open...
- 1047 Views
- 1 replies
- 0 kudos
- 0 kudos
You cannot do this from the notebooks. But you can do it via the SQL editor as shown below.
- 0 kudos
- 2907 Views
- 3 replies
- 2 kudos
Resolved! Error updating tables in DLT
I'm working on a Delta Live Tables (DLT) pipeline in Databricks Serverless mode.I receive a stream of data from Event Hubs, where each incoming record contains a unique identifier (uuid) along with some attributes (code1, code2).My goal is to update ...
- 2907 Views
- 3 replies
- 2 kudos
- 2 kudos
All the tables that DLT writes to or updates needs to be managed by DLT. The reason is that these tables are streaming tables and hence DLT needs to manage the checkpointing. It also does the optimization for such tables. So in your scenario, you can...
- 2 kudos
- 2588 Views
- 4 replies
- 1 kudos
How can I use multiple schema in DLT?
Hi I would like to use multiple schema as destination in dlt but currently I can just give single unity schema . I have my tables of multi hop in different schema.
- 2588 Views
- 4 replies
- 1 kudos
- 1 kudos
@Hariharan49 'Direct Publishing Mode' Public Preview is now live on all production regions. This feature will allow you to write to multiple schemas & catalogs from the same pipeline.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
