- 16531 Views
- 14 replies
- 5 kudos
toPandas() causes IndexOutOfBoundsException in Apache Arrow
Using DBR 10.0 When calling toPandas() the worker fails with IndexOutOfBoundsException. It seems like ArrowWriter.sizeInBytes (which looks like a proprietary method since I can't find it in OSS) calls arrow's getBufferSizeFor which fails with this ...
- 16531 Views
- 14 replies
- 5 kudos
- 5 kudos
I am also facing the same issue, I have applied the config: `spark.sql.execution.arrow.pyspark.enabled` set to `false`, but still facing the same issue. Any Idea, what's going on???. Please help me out....org.apache.spark.SparkException: Job aborted ...
- 5 kudos
- 4000 Views
- 3 replies
- 1 kudos
Resolved! Store data using client side encryption and read data using client side encryption
Hi All,I am looking for some options to add the Client side encryption feature of azure to store data in adls gen2https://learn.microsoft.com/en-us/azure/storage/blobs/client-side-encryption?tabs=javaAny help will be highly appreciatedNote: Fernet si...
- 4000 Views
- 3 replies
- 1 kudos
- 1 kudos
@Vidula Khanna​ We are going with fernet encryption as direct method is not available
- 1 kudos
- 1562 Views
- 2 replies
- 0 kudos
access databricks secretes in int script
we are trying install databricks cli on init scripts and in order to do this we need to autheticate with databricks token but it is not secure as anyone got access to cluster can get hold of this databricks token.we try to inject the secretes into se...
- 1562 Views
- 2 replies
- 0 kudos
- 0 kudos
I think you don't need to install CLI. There is a whole API available via notebook. below is example:import requests ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() host_name = ctx.tags().get("browserHostName").get() host_toke...
- 0 kudos
- 7195 Views
- 4 replies
- 11 kudos
Resolved! Pyspark learning path
Can anyone suggest to take the best series of courses offered by Databricks to learn pyspark for ETL purpose either in Databricks partner learning portal or Databricks learning portal.
- 7195 Views
- 4 replies
- 11 kudos
- 11 kudos
To learn Databricks ETL, I highy recommend videos made by Simon on that channel https://www.youtube.com/@AdvancingAnalytics
- 11 kudos
- 4274 Views
- 9 replies
- 5 kudos
I am preparing for the data analyst exam and I need as many resources as I can get to fully prepare. Hands-on labs will be welcome as well
I am preparing for the data analyst exam and I need as many resources as I can get to fully prepare. Hands-on labs will be welcome as well
- 4274 Views
- 9 replies
- 5 kudos
- 5 kudos
Hi,Kindly refer to the materials below:Videohttps://info.databricks.com/dc/kvtpV3WYob2etSFEoxuDGMYVc6afyrIMgIW50ZzIbvpUgj2uOQyz91VsFjIVPsTMDcYAQ8K0HTbFHGKunTHn_tZmFrrG7SaByl8pfwUNMIZfHhQHiMHwQEKzYSwtM9Vr6hKVl28RlEsSlOluDqaxKqoLcg8-qEwq4xtnrG8zKMEOSpQ...
- 5 kudos
- 2160 Views
- 3 replies
- 5 kudos
Databricks Cross cloud
We have service with AWS Databricks. We are doing the same replica on GCP Databricks. Here we required all the services and functionalities should be run in AWS and AWS Databricks. The only thing data should be stored on the GCP Storage. Simply funct...
- 2160 Views
- 3 replies
- 5 kudos
- 5 kudos
no, right now i don't think they are supporting this type of architecture
- 5 kudos
- 3125 Views
- 4 replies
- 4 kudos
Databricks dashboard alias URL
Can we create a alias URL for databricks dashboard ?
- 3125 Views
- 4 replies
- 4 kudos
- 4 kudos
yes it is possible , you have to create DNS for your databricks and map it to your workspace URL
- 4 kudos
- 1978 Views
- 1 replies
- 2 kudos
#00244807 and #00245872 Ticket Status - HIGH Priority
Dear @Vidula Khanna​ Vidula, Databricks team, @nadia Elsayed​ @Jose Gonzalez​ @Aden Jaxson​ What is the SLA/ETA for normal priority ticket and HIGH priority ticket? I created tickets #00244807 on 7th Dec and #00245872 but haven't received any u...
- 1978 Views
- 1 replies
- 2 kudos
- 2 kudos
you can only create high-priority tasks if you have an enterprise plan.as a normal user you can only create normal tasksif you have enterprise plan then you can escalate case .databricks team will revert you soon there.
- 2 kudos
- 5697 Views
- 1 replies
- 1 kudos
Resolved! Masking A Data Column
Is there a way to mask the data in a column in a table from specific users or user groups?
- 5697 Views
- 1 replies
- 1 kudos
- 1 kudos
yesthis doc will be helpful for you -- https://www.databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html
- 1 kudos
- 1122 Views
- 1 replies
- 0 kudos
Materials for preparing data bricks professional exam.
Hi All, Is there any book / materials for studying for data bricks professional certification ?Thank You !!!
- 1122 Views
- 1 replies
- 0 kudos
- 0 kudos
please check databricks academy,there you will find the right courses
- 0 kudos
- 1210 Views
- 2 replies
- 1 kudos
i need a regex to get whole word with parentheses
SELECT '(CC) ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST1, 'A(CC) ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST2, 'A (CC)A ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST3, 'A (CC) A ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST4, 'A ABC (CC)' REGEXP '\\b\\(CC\\)\\b' AS TES...
- 1210 Views
- 2 replies
- 1 kudos
- 1 kudos
get whole word "(CC)"I had already written the outputexpected outuput '(CC) ABC' REGEXP <<regex>> = TRUE'A(CC) ABC' REGEXP <<regex>> = FALSE'A (CC)A ABC' REGEXP <<regex>> = FALSE 'A (CC) A ABC' REGEXP <<regex>> = TRUE 'A ABC (CC)' REGEXP <<regex>> = ...
- 1 kudos
- 1051 Views
- 1 replies
- 0 kudos
- 1051 Views
- 1 replies
- 0 kudos
- 0 kudos
use python formatting
- 0 kudos
- 2630 Views
- 1 replies
- 2 kudos
execute sql server agent jobs from Databricks notebook
Is it possible to execute sql server agent job from Databricks notebook?
- 2630 Views
- 1 replies
- 2 kudos
- 2 kudos
i dont think this type of feature is available there
- 2 kudos
- 1056 Views
- 1 replies
- 3 kudos
Note book formatting
I am using DBR 10.4 LTS instance can anyone help me formatting the code.I have tried with format python error pop up with upgrade to DBR 11.2 any other alternative to this?
- 1056 Views
- 1 replies
- 3 kudos
- 3 kudos
please give us a code by that we can help you
- 3 kudos
- 2480 Views
- 1 replies
- 0 kudos
Driver restarts and job dies after 10-20 hours (Structured Streaming)
I am running a java/jar Structured Streaming job on a single node cluster (Databricks runtime 8.3). The job contains a single query which reads records from multiple Azure Event Hubs using Spark Kafka functionality and outputs results to a mssql dat...
- 2480 Views
- 1 replies
- 0 kudos
- 0 kudos
its seems that when your nodes are increasing it is seeking for init script and it is failing so you can use reserve instances for this activity instead of spot instances it will increase your overall costor alternatively, you can use depended librar...
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
