- 4183 Views
- 1 replies
- 0 kudos
Error Sent message larger than max
Hello, I'm receiving a large amount of data in a dataframe, when trying to record or display it, I receive the error below. How can I fix it, or where do I change the setting?SparkConnectGrpcException: <_MultiThreadedRendezvous of RPC that terminated...
- 4183 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod, I'm facing the same error. Can you provide the full name of the spark conf we have to change?Thank you
- 0 kudos
- 820 Views
- 0 replies
- 0 kudos
Critical/Important Spark Listener for performance tuning of spark code
Please suggest some key/important Listener which is most helpful in performance tuning of spark code.Kindly suggest how to use the listener in pyspark code to access the metrics as well for reference.
- 820 Views
- 0 replies
- 0 kudos
- 4816 Views
- 3 replies
- 2 kudos
Access Databricks API using IDP token
Hello,We have a databricks account & workspace, provided by AWS with SSO enabled. Is there any way to access databricks workspace API ( jobs/clusters, etc ) using a token retrieved from IdentityProvider ? We can access databricks workspace API with A...
- 4816 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey - Costin and Anonymous user, have you managed to get this working, do you have examples by any chance?I'm also trying something similar but I haven't been able to make it work.> authenticate and access the Databricks REST API by setting the Autho...
- 2 kudos
- 2723 Views
- 2 replies
- 0 kudos
AWS NAT (Network Address Translation) Automated On-demand Destruct / Create
Hi folks, Our company typically uses Databrick during a 12 hour block, however the AWS NAT for elastic compute is up 24 hours, and I'd rather not pay for those hours.I gather AWS lambda and cloudwatch can be used to schedule / trigger NAT destruction...
- 2723 Views
- 2 replies
- 0 kudos
- 0 kudos
For interest, this is how I ended up solving the situation, with pointers from AWS support:<< CREATE NAT >>import boto3 import logging from datetime import datetime ec2 = boto3.client('ec2') cloudwatch = boto3.client('logs') def lambda_handler(even...
- 0 kudos
- 2042 Views
- 2 replies
- 1 kudos
How to update num_workers dynamically in a job cluster
I am setting up a workflows with the UI. In the first task, a dynamic value for the next task's num_workers is calculated based on actual data size. In the subsequent task, I'd like to use this calculated num_workers to update the job cluster's defau...
- 2042 Views
- 2 replies
- 1 kudos
- 1730 Views
- 1 replies
- 1 kudos
Discrepancy in Record Count in DLT Pipeline Data Quality Tab
Hello,I have set up a DLT pipeline where I ingest data from a Kafka topic into a table. Then, I create another table that filters records from the first table. However, I'm facing an issue:When I check the Data Quality tab for the second table, it sh...
- 1730 Views
- 1 replies
- 1 kudos
- 9880 Views
- 3 replies
- 5 kudos
Resolved! Driver: Out of Memory
Hi everyone,I have a streaming job with 29 notebooks that runs continuously. Initially, I allocated 28 GB of memory to the driver, but the job failed with a "Driver Out of Memory" error after 4 hours of execution.To address this, I increased the driv...
- 9880 Views
- 3 replies
- 5 kudos
- 5 kudos
It looks like _source_cdc_time is the timestamp for when the CDC transaction occurred in your source system. This would be a good choice for a timestamp column for your watermark, since you would be deduping values according to the time the transacti...
- 5 kudos
- 15690 Views
- 1 replies
- 1 kudos
Delta table and AnalysisException: [PATH_NOT_FOUND] Path does not exist
I am performing some tests with delta tables. For each test, I write a delta table to Azure Blob Storage. Then I manually delete the delta table. After deleting the table and running my code again, I get this error: AnalysisException: [PATH_NOT_FOUN...
- 15690 Views
- 1 replies
- 1 kudos
- 1 kudos
yes it is weird , workaround for thisfiles = dbutils.fs.ls("s3 bucket or azure blob path")file_paths = [file.path for file in files]if target_path not in file_paths: dbutils.fs.mkdirs(target_path)
- 1 kudos
- 737538 Views
- 33 replies
- 5 kudos
GC Driver Error
I am using a cluster in databricks to connect to a Tableau workbook through the JDBC connector. My Tableau workbook has been unable to load due to resources not being available through the data connection. I went to look at the driver log for my clus...
- 737538 Views
- 33 replies
- 5 kudos
- 12237 Views
- 3 replies
- 7 kudos
Resolved! Efficient caching/persisting
To cache/persist an action needs to be triggered. I'm just wondering, will it make any difference if, after persisting some df, I use, for instance, take(5) instead of count()?Will it be a bit more effective, because of sending results from 5 partiti...
- 12237 Views
- 3 replies
- 7 kudos
- 7 kudos
Yes take (5) will be more efficient in some ways.When you cache or persist a DataFrame in Spark, you are instructing Spark to store the DataFrame's intermediate data in memory (or on disk, depending on the storage level). This can significantly speed...
- 7 kudos
- 2015 Views
- 2 replies
- 0 kudos
dbfs /mounts permissions with Clusters on Shared Mode / Serverless
Hi allI've used mounts based on service principals but users using shaed clusters or the new serverless they have problems with permissions to access resources on dbfs. Right now we have used clusters in single modeWhat should be the best approach to...
- 2015 Views
- 2 replies
- 0 kudos
- 3916 Views
- 0 replies
- 0 kudos
how to migrate features from azure databricks workspace to gcp
I’m in the process of migrating feature tables from Azure Databricks to GCP Databricks and am having trouble listing all feature tables from Azure Databricks.I’ve tried using the FeatureStoreClient API, but it doesn’t have a function to list all feat...
- 3916 Views
- 0 replies
- 0 kudos
- 8154 Views
- 6 replies
- 3 kudos
Resolved! Is Concurrent Writes from multiple databricks clusters to same delta table on S3 Supported?
Does databricks have support for writing to same Delta Table from multiple clusters concurrently. I am specifically interested to know if there is any solution for https://github.com/delta-io/delta/issues/41 implemented in databricks OR if you have a...
- 8154 Views
- 6 replies
- 3 kudos
- 3 kudos
Please note, the issue noted above [Storage System] Support for AWS S3 (multiple clusters/drivers/JVMs) is for Delta Lake OSS. As noted in this issue as well as Issue 324, as of this writing, S3 lacks putIfAbsent transactional consistency. For Del...
- 3 kudos
- 3643 Views
- 2 replies
- 1 kudos
Resolved! Ingesting logs from Databricks (GCP) to Azure log Analytics
Hi everyone, I wanted to ask if there is any way through which we can ingest logs from GCP databricks to azure log analytics in store-sync fashion. Meaning we will save logs into some cloud bucket lets say, then from there we should be able to send l...
- 3643 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod ,Thanks for help. We decided to develop our own library for logging to azure log analytics. We used buffer for this. We are currently on timer based logs but in future versions we wanted to move to memory based.Thanks,Nikhil
- 1 kudos
- 4977 Views
- 0 replies
- 0 kudos
Federated Learning for Decentralized, Secure Model Training
Federated learning allows you to train machine learning models on decentralized data while ensuring data privacy and security by storing data on local devices and only sharing model updates. This approach assures that raw data never leaves its source...
- 4977 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
