- 1130 Views
- 1 replies
- 0 kudos
Sqoop Migration best practices
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-SQOOP to Databricks?
- 1130 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
- 0 kudos
- 635 Views
- 1 replies
- 0 kudos
OOZE Jobs migration to databricks
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-ooze jobs to Databricks?
- 635 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources on Hadoop migration: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-...
- 0 kudos
- 1270 Views
- 1 replies
- 0 kudos
HDFS to Databricks
Hi ,Could you please share with us the approach and best practices for migrating from hadoop-HDFS to Databricks?
- 1270 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can try checking the below resources: https://www.databricks.com/resources/ebook/migration-guide-hadoop-to-databricks https://www.databricks.com/solutions/migration/hadoop https://www.databricks.com/blog/2021/08/06/5-key-steps-to-successfull...
- 0 kudos
- 914 Views
- 1 replies
- 0 kudos
Notebook runs with error when run as a job
I am using a notebook to copy over my database on a schedule (I had no success connecting through the Data Explorer UI). When I run the notebook on its own, it works. When I run it as a scheduled job, I get this error. org.apache.spark.SparkSQLExcept...
- 914 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, the error code is minimal, could you please post the whole error if that is possible? Also please tag @Debayan​ with your next response which will notify me, Thank you!
- 0 kudos
- 7207 Views
- 1 replies
- 0 kudos
Resolved! Recover Account Owner
Need help recovering account owner.Problem: Account owner cannot sign in with its password after SSO was configured. Account owner is a DL for team ownership so it doesn't have an AWS account and can't configure in AD group since it has "+" in the em...
- 7207 Views
- 1 replies
- 0 kudos
- 0 kudos
Resolved by temporarily disabling SSO with Active directory that wasn't allowing an email to be created with "+"
- 0 kudos
- 2504 Views
- 3 replies
- 1 kudos
DLT Pipeline unable to find custom Libraries/Wheel packages
We have our DLT pipeline and we need to import our custom libraries packaged in wheel files.We are on Azure DBX and we are using Az DevOps CI/CD to build and deploy the wheel packages on our DBX environment. In the top of our DLT notebook we are impo...
- 2504 Views
- 3 replies
- 1 kudos
- 1 kudos
"context_based_upload_for_execute": truein projects.json allowed the code to run - but ended withRuntimeError: Cannot start a remote Spark session because there is a regular Spark session already running.
- 1 kudos
- 1682 Views
- 1 replies
- 0 kudos
Will the cells in the notebook keep running even if the browser is closed?
The Execute ML Model Pipeline celll has finished running, it took 2.27 days to finish running. However, the code in the following cell called Process JSON Output needs to take a very long time again to run. Can I simply close the browser and shut dow...
- 1682 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I have just tested it internally, even if the browser is closed the notebook keeps on running. You can start with a quick job to test it. Also, please tag @Debayan with your next response so that I will get notified.
- 0 kudos
- 1209 Views
- 2 replies
- 0 kudos
Is it a bug in DEEP CLONE?
Hi,I'm trying to modify a delta table using following approach:Shallow clone of the table (source_table)Modification of the the clone (clonned_table)Deep clone of the modified table to the source table.Source delta table has 26 752 rows. Current Delt...
- 1209 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @norbitek, The issue you are experiencing is likely due to the nature of "shallow" cloning. A "shallow" clone in Delta Lake does not copy the data files to the clone target. Instead, it creates pointers to the existing data files. This means that ...
- 0 kudos
- 7456 Views
- 2 replies
- 2 kudos
Resolved! Error Row Column access control Demo Unity Catalog
I have this error when I execute the Table ACL & Row and Column Level Security With Unity Catalog demo :Error in SQL statement: UnityCatalogServiceException: [RequestId=9644499b-3555-43d0-a2e7-0fdc29ea85ce ErrorClass=NOT_IMPLEMENTED.NOT_IMPLEMENTED] ...
- 7456 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @EmmaP, The error you are experiencing, "UnityCatalogServiceException: [RequestId=9644499b-3555-43d0-a2e7-0fdc29ea85ce ErrorClass=NOT_IMPLEMENTED.NOT_IMPLEMENTED] The UpdateTable row and column access policy operation is not currently implemented"...
- 2 kudos
- 1077 Views
- 1 replies
- 1 kudos
Unable to change/cast column datatype using Delta IO
I have created a delta table using Delta IO library, with following detailsTable Name: EmployeeColumns {Id Integer, name String, gender String, Salary decimal(5,2)}Now I want to upcast the salary from decimal(5,2) to decimal(10,4). If I use delta IO ...
- 1077 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Srikanthn , Yes, you can change the datatype of a column in a Delta table without losing the delta log. The delta log provides transaction history which is essential for time travel in delta tables. Here is the method to change the datatype of a...
- 1 kudos
- 2160 Views
- 1 replies
- 0 kudos
Update values in DataFrame with values from another DataFrame
Hi, I have to data sources: "admin table" which contains active and inactive employees and I have "hr table" which contains only active employees.I need to update admin table with data from hr table:You can see that I need to update employee with num...
- 2160 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AleksandraFrolo , To update your "admin" table with data from the "hr" table, you can use the MERGE INTO statement provided by Databricks Delta. This operation is known as an upsert (update or insert). Here's an example of how you can do it: sqlM...
- 0 kudos
- 1037 Views
- 1 replies
- 0 kudos
Connecting to databricks ipython kernel from VSCode
I'd like to run python notebooks (.ipynb) from VSCode connecting to the ipython kernel from databricks. I have already connected to an execution cluster from VSCode and am able to run python scripts (.py files) and see the output on my local console....
- 1037 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rotate_employee, I recommend checking Databricks or VSCode documentation or file a support ticket for further assistance.
- 0 kudos
- 2530 Views
- 6 replies
- 0 kudos
Need help migrating company customer and partner academy accounts to work properly
Hi, originally I accidentally made a customer academy account with my company that is a databricks partner. Then I made an account using my personal email and listed my company email as the partner email for the partner academy account. that account ...
- 2530 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Maria_fed Thanks again, i have assigned your case to my colleague and you should hearing from them soon. Regards, Akshay
- 0 kudos
- 1108 Views
- 2 replies
- 1 kudos
Getting FileNotFoundException while using cloudFiles
Hi,Following is the code i am using the ingest the data incrementally (weekly).val ssdf = spark.readStream.schema(schema) .format("cloudFiles").option("cloudFiles.format", "parquet").load(sourceUrl).filter(criteriaFilter)val transformedDf = ssdf.tran...
- 1108 Views
- 2 replies
- 1 kudos
- 843 Views
- 1 replies
- 0 kudos
RBAC, Security & Privacy controls
Could you please share us best practices on implementation of RBAC, Security & Privacy controls in Databricks
- 843 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please check on https://docs.databricks.com/en/lakehouse-architecture/security-compliance-and-privacy/best-practices.html and see if this helping? Also, please tag @Debayan with your next comment which will notify me. Thanks!
- 0 kudos
