- 14762 Views
- 2 replies
- 0 kudos
S3 write to bucket - best performance tips
I'm writing big dataframes into deltas in s3 buckets. df.write\ .format("delta")\ .mode("append")\ .partitionBy(partitionColumns)\ .option("mergeSchema", "true")\ .save(target...
- 14762 Views
- 2 replies
- 0 kudos
- 0 kudos
@Pablo (Ariel) :There are several ways to improve the performance of writing data to S3 using Spark. Here are some tips and recommendations:Increase the size of the write buffer: By default, Spark writes data in 1 MB batches. You can increase the si...
- 0 kudos
- 9317 Views
- 1 replies
- 2 kudos
Resolved! Best Practices for PowerBI Connectivity w/ Partner Connect. Access Token w/ Service Principal, Databricks Username w/ Service account, or OAuth?
I'm aware all are possible methods but are all equal? Or is the matter trivial? Thank you so much!
- 9317 Views
- 1 replies
- 2 kudos
- 2 kudos
@Will Heyer :The best method for Power BI connectivity with Partner Connect depends on your specific use case and requirements. Here are some factors to consider for each method:Access Token with Service Principal: This method uses a client ID and s...
- 2 kudos
- 17870 Views
- 1 replies
- 4 kudos
Resolved! How to use multi-cursor and rectangular selection for notebooks and query editor in Linux ?
The documentation explains how to use multicursor in notebooks. However, it only says it for Windows and MacOS. The Windows way would work in Linux (Ubuntu) up to a few days ago but it does not work now anymore.
- 17870 Views
- 1 replies
- 4 kudos
- 4 kudos
@Davide Cagnoni :Multicursor support in Databricks notebooks is implemented using the Ace editor, which is a web-based code editor. Therefore, the behavior of multicursor support may depend on the specific browser and operating system you are using....
- 4 kudos
- 2862 Views
- 1 replies
- 1 kudos
Resolved! Clusters API 2.0 - Unable to execute cluster events api
Details: I keep getting "Missing required field: cluster_id" even though you can see it is supplied. Is this a bug? or I am missing something? I am testing this in postman. Error: {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fi...
- 2862 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi guysI'm sorry to hear that the Clusters API 2.0 and cluster event execution are giving you trouble. I advise getting in touch with the support staff for guidance on quickly fixing the problem.
- 1 kudos
- 2614 Views
- 1 replies
- 1 kudos
Resolved! DLT best practices
Hi Team,Could you please recommend the best practices to implement the delta live tables?Regards,Phanindra
- 2614 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Phani, what exactly are you looking for with best practices? At a high level:Always provide an external storage location (S3, ADLS, GCS) for your pipelineUse Auto Scaling! Python imports can be leverage to reuse code With regards to providing a st...
- 1 kudos
- 7350 Views
- 1 replies
- 2 kudos
Resolved! Azure Databricks PATs expire even before validity
Hi all,we have this issue in our environment - even thought we give 365 days validity for Databricks PATS generation, the PATs expire every now and then. Is there any problem with the command we use : curl --location --request POST 'https://<<HOST_NA...
- 7350 Views
- 1 replies
- 2 kudos
- 2 kudos
@NOOR BASHA SHAIK It looks you are providing 365 days, can you please post your response. if you won't provide any lifetime then it should be valid indefinitely. can you please add 90 days validity and test
- 2 kudos
- 1626 Views
- 1 replies
- 1 kudos
API to get Databricks Status AWS.
Hi, Do you have an api endpoint to call to get the databricks status for AWS?Thanks,
- 1626 Views
- 1 replies
- 1 kudos
- 1 kudos
@Chinu Lee you have webhook/slack that can be used to fetch status https://docs.databricks.com/resources/status.html#webhookare you specifically looking for your account workspace/above one
- 1 kudos
- 2196 Views
- 1 replies
- 3 kudos
Create (account wide) groups without account admin permissions
The use case is quite simple: each environment - databricks workspace (prod, test, dev) will be created by a separate service principal (which for isolation purpose should not have account wide admin permission) with terraform, but will belong to the...
- 2196 Views
- 1 replies
- 3 kudos
- 3 kudos
Another thing would be also to assign workspace to a metastore without account admin permission - for similar reason.
- 3 kudos
- 8611 Views
- 6 replies
- 2 kudos
Resolved! Delta Sharing - Unity Catalog difference
Delta Sharing and Unity catalog both have elements of data sharing. Can you please explain when one would use Delta sharing vs Unity Catalog?
- 8611 Views
- 6 replies
- 2 kudos
- 2 kudos
Based on the Databricks reply from the post below: "Unity Catalog does not currently support separating data by workspace or Azure subscription. As you noted, data from all catalogs within a region can be accessed by any workspace within that region,...
- 2 kudos
- 8857 Views
- 7 replies
- 9 kudos
Can't able to run following queries
I cannot able to run following queriesALTER TABLE iot_events ADD ATTRIBUTE pii ON emailALTER TABLE users ADD ATTRIBUTE pii ON phoneGRANT SELECT ON DATABASE iot_data HAVING ATTRIBUTE NOT IN (pii) TO product_managersand GRANT SELECT ON iot_events TO ...
- 8857 Views
- 7 replies
- 9 kudos
- 9 kudos
@Kaniz Fatma can anyone from data Bircks help on why attribute-based access control function is not working in unity catalog @Rahul Mishra below commands ALTER TABLE iot_events ADD ATTRIBUTE pii ON emailALTER TABLE users ADD ATTRIBUTE pii ON phoneG...
- 9 kudos
- 5831 Views
- 1 replies
- 2 kudos
Resolved! The Jupiter note-book doesn't update imports after updating the .py file
Please help. Here's an example:I have one .py file and one .ipynb, and the .py file contains the test function, but after adding the new function test1, it doesn't appear in .ipynb. Even after re-running the .py file and reimporting it in .ipynb. How...
- 5831 Views
- 1 replies
- 2 kudos
- 2 kudos
%load_ext autoreload%autoreload 2
- 2 kudos
- 2269 Views
- 1 replies
- 0 kudos
clickfortranslation.com
What are the Different Types of Industry Specific Translation Services Manual ?GET TO KNOW MORE CLICK THE LINK BELOW.https://clickfortranslation.com/manual-translation.php
- 2269 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you provide all types of industry-specific translation services? I'm asking because many translation companies pretend to provide all industry-specific translation services, but the final result is far from being a qualitative and professional one...
- 0 kudos
- 3293 Views
- 2 replies
- 2 kudos
Does anyone run Databricks jobs using Docker + ARM (Graviton) instances?
Graviton instances do not support Container services on paper (https://docs.databricks.com/clusters/graviton.html#unsupported-features) but if you try to build Docker ARM image and run it on Graviton, it will work. Does anyone use this combination in...
- 3293 Views
- 2 replies
- 2 kudos
- 2 kudos
Graviton is not supported by Databricks Container Services. How are you planning to run it on Databricks? Please tag @Debayan with your next comment so that I will get notified. Thank you!
- 2 kudos
- 10400 Views
- 2 replies
- 2 kudos
Resolved! How to resolve SAT driver errors
I was able to follow the SAT setup instructions, but ran into the same error whether I ran it "manually" or via terraform. The initialization seemed to run fine. Can anyone suggest any steps to troubleshoot this?
- 10400 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks - I also spoke with Arun, who was very helpful. Our databricks admin users all require an okta login, which is causing the error. We're looking into a "break glass" admin user for this purpose.
- 2 kudos
- 7915 Views
- 5 replies
- 4 kudos
Is there a way to clone job cluster or edit cluster using JSON?
I've created workflow job (let say job A) and set up job cluster configuration for it.Now I want to create another workflow job (job B) but use almost the same settings for job cluster.I can see cluster settings in JSON (for both jobs) but I can't ed...
- 7915 Views
- 5 replies
- 4 kudos
- 4 kudos
Also you can use terraform exporter with -match flag to get a .tf definition for a job A. Once initialized, you can define job B.Another option is to use dbx
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
